2023-04-18 18:53:33,100 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc 2023-04-18 18:53:33,115 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-18 18:53:33,133 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-18 18:53:33,133 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/cluster_a8fdfc0f-6800-c939-7c96-ea4c4c9b2e3f, deleteOnExit=true 2023-04-18 18:53:33,134 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-18 18:53:33,134 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/test.cache.data in system properties and HBase conf 2023-04-18 18:53:33,135 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/hadoop.tmp.dir in system properties and HBase conf 2023-04-18 18:53:33,135 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/hadoop.log.dir in system properties and HBase conf 2023-04-18 18:53:33,136 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-18 18:53:33,136 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-18 18:53:33,136 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-18 18:53:33,259 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-18 18:53:33,659 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-18 18:53:33,663 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-18 18:53:33,663 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-18 18:53:33,664 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-18 18:53:33,664 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-18 18:53:33,664 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-18 18:53:33,665 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-18 18:53:33,665 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-18 18:53:33,665 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-18 18:53:33,665 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-18 18:53:33,666 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/nfs.dump.dir in system properties and HBase conf 2023-04-18 18:53:33,666 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/java.io.tmpdir in system properties and HBase conf 2023-04-18 18:53:33,666 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-18 18:53:33,666 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-18 18:53:33,667 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-18 18:53:34,287 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-18 18:53:34,292 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-18 18:53:34,595 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-18 18:53:34,768 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-18 18:53:34,785 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-18 18:53:34,826 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-18 18:53:34,858 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/java.io.tmpdir/Jetty_localhost_46243_hdfs____.xpv3l3/webapp 2023-04-18 18:53:35,008 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46243 2023-04-18 18:53:35,061 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-18 18:53:35,062 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-18 18:53:35,493 WARN [Listener at localhost/40835] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-18 18:53:35,616 WARN [Listener at localhost/40835] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-18 18:53:35,635 WARN [Listener at localhost/40835] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-18 18:53:35,643 INFO [Listener at localhost/40835] log.Slf4jLog(67): jetty-6.1.26 2023-04-18 18:53:35,649 INFO [Listener at localhost/40835] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/java.io.tmpdir/Jetty_localhost_33747_datanode____1ifo8x/webapp 2023-04-18 18:53:35,783 INFO [Listener at localhost/40835] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33747 2023-04-18 18:53:36,146 WARN [Listener at localhost/36951] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-18 18:53:36,763 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x26c3d3fa77260a2a: Processing first storage report for DS-d78a5de1-2028-488e-9841-5b8299adb3db from datanode 309c0e06-eb1a-47ea-b39b-d28d4ae1a8b8 2023-04-18 18:53:36,764 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x26c3d3fa77260a2a: from storage DS-d78a5de1-2028-488e-9841-5b8299adb3db node DatanodeRegistration(127.0.0.1:38449, datanodeUuid=309c0e06-eb1a-47ea-b39b-d28d4ae1a8b8, infoPort=38677, infoSecurePort=0, ipcPort=36951, storageInfo=lv=-57;cid=testClusterID;nsid=332790490;c=1681844014371), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-18 18:53:36,765 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x26c3d3fa77260a2a: Processing first storage report for DS-96e050c8-82eb-40c0-86f0-55e2aea9e543 from datanode 309c0e06-eb1a-47ea-b39b-d28d4ae1a8b8 2023-04-18 18:53:36,765 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x26c3d3fa77260a2a: from storage DS-96e050c8-82eb-40c0-86f0-55e2aea9e543 node DatanodeRegistration(127.0.0.1:38449, datanodeUuid=309c0e06-eb1a-47ea-b39b-d28d4ae1a8b8, infoPort=38677, infoSecurePort=0, ipcPort=36951, storageInfo=lv=-57;cid=testClusterID;nsid=332790490;c=1681844014371), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-18 18:53:36,778 DEBUG [Listener at localhost/36951] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc 2023-04-18 18:53:36,869 INFO [Listener at localhost/36951] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/cluster_a8fdfc0f-6800-c939-7c96-ea4c4c9b2e3f/zookeeper_0, clientPort=65091, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/cluster_a8fdfc0f-6800-c939-7c96-ea4c4c9b2e3f/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/cluster_a8fdfc0f-6800-c939-7c96-ea4c4c9b2e3f/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-18 18:53:36,885 INFO [Listener at localhost/36951] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=65091 2023-04-18 18:53:36,896 INFO [Listener at localhost/36951] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-18 18:53:36,899 INFO [Listener at localhost/36951] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-18 18:53:37,579 INFO [Listener at localhost/36951] util.FSUtils(471): Created version file at hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab with version=8 2023-04-18 18:53:37,579 INFO [Listener at localhost/36951] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/hbase-staging 2023-04-18 18:53:37,892 INFO [Listener at localhost/36951] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-18 18:53:38,366 INFO [Listener at localhost/36951] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-18 18:53:38,399 INFO [Listener at localhost/36951] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-18 18:53:38,399 INFO [Listener at localhost/36951] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-18 18:53:38,400 INFO [Listener at localhost/36951] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-18 18:53:38,400 INFO [Listener at localhost/36951] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-18 18:53:38,400 INFO [Listener at localhost/36951] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-18 18:53:38,559 INFO [Listener at localhost/36951] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-18 18:53:38,634 DEBUG [Listener at localhost/36951] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-18 18:53:38,746 INFO [Listener at localhost/36951] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39391 2023-04-18 18:53:38,758 INFO [Listener at localhost/36951] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-18 18:53:38,761 INFO [Listener at localhost/36951] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-18 18:53:38,787 INFO [Listener at localhost/36951] zookeeper.RecoverableZooKeeper(93): Process identifier=master:39391 connecting to ZooKeeper ensemble=127.0.0.1:65091 2023-04-18 18:53:38,829 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:393910x0, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-18 18:53:38,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:39391-0x101989f00970000 connected 2023-04-18 18:53:38,860 DEBUG [Listener at localhost/36951] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-18 18:53:38,861 DEBUG [Listener at localhost/36951] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-18 18:53:38,864 DEBUG [Listener at localhost/36951] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-18 18:53:38,872 DEBUG [Listener at localhost/36951] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39391 2023-04-18 18:53:38,873 DEBUG [Listener at localhost/36951] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39391 2023-04-18 18:53:38,873 DEBUG [Listener at localhost/36951] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39391 2023-04-18 18:53:38,874 DEBUG [Listener at localhost/36951] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39391 2023-04-18 18:53:38,874 DEBUG [Listener at localhost/36951] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39391 2023-04-18 18:53:38,880 INFO [Listener at localhost/36951] master.HMaster(444): hbase.rootdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab, hbase.cluster.distributed=false 2023-04-18 18:53:38,953 INFO [Listener at localhost/36951] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-18 18:53:38,954 INFO [Listener at localhost/36951] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-18 18:53:38,954 INFO [Listener at localhost/36951] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-18 18:53:38,954 INFO [Listener at localhost/36951] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-18 18:53:38,954 INFO [Listener at localhost/36951] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-18 18:53:38,954 INFO [Listener at localhost/36951] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-18 18:53:38,959 INFO [Listener at localhost/36951] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-18 18:53:38,963 INFO [Listener at localhost/36951] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34053 2023-04-18 18:53:38,965 INFO [Listener at localhost/36951] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-18 18:53:38,971 DEBUG [Listener at localhost/36951] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-18 18:53:38,972 INFO [Listener at localhost/36951] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-18 18:53:38,975 INFO [Listener at localhost/36951] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-18 18:53:38,977 INFO [Listener at localhost/36951] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34053 connecting to ZooKeeper ensemble=127.0.0.1:65091 2023-04-18 18:53:38,980 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:340530x0, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-18 18:53:38,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34053-0x101989f00970001 connected 2023-04-18 18:53:38,982 DEBUG [Listener at localhost/36951] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-18 18:53:38,983 DEBUG [Listener at localhost/36951] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-18 18:53:38,984 DEBUG [Listener at localhost/36951] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-18 18:53:38,985 DEBUG [Listener at localhost/36951] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34053 2023-04-18 18:53:38,985 DEBUG [Listener at localhost/36951] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34053 2023-04-18 18:53:38,988 DEBUG [Listener at localhost/36951] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34053 2023-04-18 18:53:38,992 DEBUG [Listener at localhost/36951] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34053 2023-04-18 18:53:38,993 DEBUG [Listener at localhost/36951] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34053 2023-04-18 18:53:38,996 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,39391,1681844017730 2023-04-18 18:53:39,009 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-18 18:53:39,010 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,39391,1681844017730 2023-04-18 18:53:39,036 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-18 18:53:39,036 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-18 18:53:39,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-18 18:53:39,038 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-18 18:53:39,039 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,39391,1681844017730 from backup master directory 2023-04-18 18:53:39,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-18 18:53:39,043 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,39391,1681844017730 2023-04-18 18:53:39,043 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-18 18:53:39,044 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-18 18:53:39,044 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,39391,1681844017730 2023-04-18 18:53:39,047 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-18 18:53:39,048 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-18 18:53:39,146 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/hbase.id with ID: 3daa6921-39d8-458a-99be-1454abe1551f 2023-04-18 18:53:39,195 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-18 18:53:39,210 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-18 18:53:39,252 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x68e6aadc to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:53:39,285 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@251497e1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:53:39,313 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-18 18:53:39,315 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-18 18:53:39,335 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-18 18:53:39,335 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-18 18:53:39,337 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:53:39,341 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:53:39,343 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-18 18:53:39,378 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/MasterData/data/master/store-tmp 2023-04-18 18:53:39,406 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 18:53:39,406 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-18 18:53:39,406 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-18 18:53:39,407 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-18 18:53:39,407 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-18 18:53:39,407 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-18 18:53:39,407 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-18 18:53:39,407 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-18 18:53:39,409 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/MasterData/WALs/jenkins-hbase4.apache.org,39391,1681844017730 2023-04-18 18:53:39,431 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39391%2C1681844017730, suffix=, logDir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/MasterData/WALs/jenkins-hbase4.apache.org,39391,1681844017730, archiveDir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/MasterData/oldWALs, maxLogs=10 2023-04-18 18:53:39,502 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38449,DS-d78a5de1-2028-488e-9841-5b8299adb3db,DISK] 2023-04-18 18:53:39,512 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:53:39,577 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/MasterData/WALs/jenkins-hbase4.apache.org,39391,1681844017730/jenkins-hbase4.apache.org%2C39391%2C1681844017730.1681844019441 2023-04-18 18:53:39,580 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38449,DS-d78a5de1-2028-488e-9841-5b8299adb3db,DISK]] 2023-04-18 18:53:39,581 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-18 18:53:39,581 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 18:53:39,585 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-18 18:53:39,586 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-18 18:53:39,649 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-18 18:53:39,657 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-18 18:53:39,684 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-18 18:53:39,698 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:53:39,705 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-18 18:53:39,707 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-18 18:53:39,727 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-18 18:53:39,731 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-18 18:53:39,733 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70641125, jitterRate=0.052634790539741516}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-18 18:53:39,733 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-18 18:53:39,734 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-18 18:53:39,757 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-18 18:53:39,757 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-18 18:53:39,761 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-18 18:53:39,763 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-18 18:53:39,803 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 39 msec 2023-04-18 18:53:39,803 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-18 18:53:39,833 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-18 18:53:39,840 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-18 18:53:39,865 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-18 18:53:39,869 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-18 18:53:39,871 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-18 18:53:39,877 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-18 18:53:39,881 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-18 18:53:39,884 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-18 18:53:39,886 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-18 18:53:39,886 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-18 18:53:39,899 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-18 18:53:39,904 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-18 18:53:39,904 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-18 18:53:39,904 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-18 18:53:39,904 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,39391,1681844017730, sessionid=0x101989f00970000, setting cluster-up flag (Was=false) 2023-04-18 18:53:39,921 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-18 18:53:39,926 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-18 18:53:39,928 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39391,1681844017730 2023-04-18 18:53:39,935 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-18 18:53:39,942 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-18 18:53:39,944 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39391,1681844017730 2023-04-18 18:53:39,948 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.hbase-snapshot/.tmp 2023-04-18 18:53:39,999 INFO [RS:0;jenkins-hbase4:34053] regionserver.HRegionServer(952): ClusterId : 3daa6921-39d8-458a-99be-1454abe1551f 2023-04-18 18:53:40,005 DEBUG [RS:0;jenkins-hbase4:34053] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-18 18:53:40,012 DEBUG [RS:0;jenkins-hbase4:34053] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-18 18:53:40,012 DEBUG [RS:0;jenkins-hbase4:34053] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-18 18:53:40,016 DEBUG [RS:0;jenkins-hbase4:34053] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-18 18:53:40,017 DEBUG [RS:0;jenkins-hbase4:34053] zookeeper.ReadOnlyZKClient(139): Connect 0x1ab7257f to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:53:40,023 DEBUG [RS:0;jenkins-hbase4:34053] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1470c859, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:53:40,024 DEBUG [RS:0;jenkins-hbase4:34053] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@104f4e0f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-18 18:53:40,049 DEBUG [RS:0;jenkins-hbase4:34053] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:34053 2023-04-18 18:53:40,053 INFO [RS:0;jenkins-hbase4:34053] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-18 18:53:40,053 INFO [RS:0;jenkins-hbase4:34053] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-18 18:53:40,053 DEBUG [RS:0;jenkins-hbase4:34053] regionserver.HRegionServer(1023): About to register with Master. 2023-04-18 18:53:40,056 INFO [RS:0;jenkins-hbase4:34053] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,39391,1681844017730 with isa=jenkins-hbase4.apache.org/172.31.14.131:34053, startcode=1681844018952 2023-04-18 18:53:40,073 DEBUG [RS:0;jenkins-hbase4:34053] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-18 18:53:40,076 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-18 18:53:40,089 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-18 18:53:40,089 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-18 18:53:40,089 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-18 18:53:40,089 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-18 18:53:40,089 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-18 18:53:40,089 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 18:53:40,090 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-18 18:53:40,090 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 18:53:40,097 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681844050097 2023-04-18 18:53:40,100 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-18 18:53:40,103 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-18 18:53:40,103 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-18 18:53:40,112 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-18 18:53:40,115 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-18 18:53:40,124 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-18 18:53:40,125 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-18 18:53:40,125 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-18 18:53:40,126 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-18 18:53:40,127 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-18 18:53:40,130 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-18 18:53:40,133 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-18 18:53:40,134 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-18 18:53:40,140 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-18 18:53:40,140 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-18 18:53:40,143 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52149, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-18 18:53:40,146 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681844020144,5,FailOnTimeoutGroup] 2023-04-18 18:53:40,147 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681844020147,5,FailOnTimeoutGroup] 2023-04-18 18:53:40,147 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-18 18:53:40,147 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-18 18:53:40,149 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-18 18:53:40,150 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-18 18:53:40,156 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-18 18:53:40,158 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-18 18:53:40,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:40,158 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab 2023-04-18 18:53:40,177 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 18:53:40,181 DEBUG [RS:0;jenkins-hbase4:34053] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab 2023-04-18 18:53:40,181 DEBUG [RS:0;jenkins-hbase4:34053] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:40835 2023-04-18 18:53:40,181 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-18 18:53:40,182 DEBUG [RS:0;jenkins-hbase4:34053] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-18 18:53:40,185 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/info 2023-04-18 18:53:40,187 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-18 18:53:40,189 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-18 18:53:40,189 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:53:40,190 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-18 18:53:40,190 DEBUG [RS:0;jenkins-hbase4:34053] zookeeper.ZKUtil(162): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:40,191 WARN [RS:0;jenkins-hbase4:34053] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-18 18:53:40,192 INFO [RS:0;jenkins-hbase4:34053] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-18 18:53:40,193 DEBUG [RS:0;jenkins-hbase4:34053] regionserver.HRegionServer(1947): logDir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/WALs/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:40,195 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,34053,1681844018952] 2023-04-18 18:53:40,196 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/rep_barrier 2023-04-18 18:53:40,197 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-18 18:53:40,198 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:53:40,199 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-18 18:53:40,202 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/table 2023-04-18 18:53:40,203 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-18 18:53:40,204 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:53:40,204 DEBUG [RS:0;jenkins-hbase4:34053] zookeeper.ZKUtil(162): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:40,205 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740 2023-04-18 18:53:40,206 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740 2023-04-18 18:53:40,216 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 18:53:40,217 DEBUG [RS:0;jenkins-hbase4:34053] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-18 18:53:40,220 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-18 18:53:40,225 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-18 18:53:40,226 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60402020, jitterRate=-0.09993976354598999}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 18:53:40,227 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-18 18:53:40,227 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-18 18:53:40,227 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-18 18:53:40,227 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-18 18:53:40,227 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-18 18:53:40,227 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-18 18:53:40,228 INFO [RS:0;jenkins-hbase4:34053] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-18 18:53:40,228 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-18 18:53:40,229 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-18 18:53:40,236 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-18 18:53:40,236 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-18 18:53:40,246 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-18 18:53:40,259 INFO [RS:0;jenkins-hbase4:34053] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-18 18:53:40,265 INFO [RS:0;jenkins-hbase4:34053] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-18 18:53:40,265 INFO [RS:0;jenkins-hbase4:34053] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-18 18:53:40,266 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-18 18:53:40,267 INFO [RS:0;jenkins-hbase4:34053] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-18 18:53:40,270 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-18 18:53:40,275 INFO [RS:0;jenkins-hbase4:34053] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-18 18:53:40,276 DEBUG [RS:0;jenkins-hbase4:34053] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 18:53:40,276 DEBUG [RS:0;jenkins-hbase4:34053] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 18:53:40,276 DEBUG [RS:0;jenkins-hbase4:34053] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 18:53:40,277 DEBUG [RS:0;jenkins-hbase4:34053] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 18:53:40,277 DEBUG [RS:0;jenkins-hbase4:34053] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 18:53:40,277 DEBUG [RS:0;jenkins-hbase4:34053] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-18 18:53:40,277 DEBUG [RS:0;jenkins-hbase4:34053] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 18:53:40,277 DEBUG [RS:0;jenkins-hbase4:34053] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 18:53:40,277 DEBUG [RS:0;jenkins-hbase4:34053] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 18:53:40,278 DEBUG [RS:0;jenkins-hbase4:34053] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 18:53:40,284 INFO [RS:0;jenkins-hbase4:34053] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-18 18:53:40,284 INFO [RS:0;jenkins-hbase4:34053] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-18 18:53:40,284 INFO [RS:0;jenkins-hbase4:34053] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-18 18:53:40,301 INFO [RS:0;jenkins-hbase4:34053] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-18 18:53:40,303 INFO [RS:0;jenkins-hbase4:34053] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34053,1681844018952-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-18 18:53:40,319 INFO [RS:0;jenkins-hbase4:34053] regionserver.Replication(203): jenkins-hbase4.apache.org,34053,1681844018952 started 2023-04-18 18:53:40,319 INFO [RS:0;jenkins-hbase4:34053] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,34053,1681844018952, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:34053, sessionid=0x101989f00970001 2023-04-18 18:53:40,320 DEBUG [RS:0;jenkins-hbase4:34053] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-18 18:53:40,320 DEBUG [RS:0;jenkins-hbase4:34053] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:40,320 DEBUG [RS:0;jenkins-hbase4:34053] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34053,1681844018952' 2023-04-18 18:53:40,320 DEBUG [RS:0;jenkins-hbase4:34053] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:40,321 DEBUG [RS:0;jenkins-hbase4:34053] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:40,322 DEBUG [RS:0;jenkins-hbase4:34053] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-18 18:53:40,322 DEBUG [RS:0;jenkins-hbase4:34053] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-18 18:53:40,322 DEBUG [RS:0;jenkins-hbase4:34053] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:40,322 DEBUG [RS:0;jenkins-hbase4:34053] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34053,1681844018952' 2023-04-18 18:53:40,322 DEBUG [RS:0;jenkins-hbase4:34053] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-18 18:53:40,322 DEBUG [RS:0;jenkins-hbase4:34053] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-18 18:53:40,323 DEBUG [RS:0;jenkins-hbase4:34053] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-18 18:53:40,323 INFO [RS:0;jenkins-hbase4:34053] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-18 18:53:40,323 INFO [RS:0;jenkins-hbase4:34053] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-18 18:53:40,422 DEBUG [jenkins-hbase4:39391] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-18 18:53:40,425 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34053,1681844018952, state=OPENING 2023-04-18 18:53:40,435 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-18 18:53:40,437 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-18 18:53:40,438 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-18 18:53:40,438 INFO [RS:0;jenkins-hbase4:34053] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34053%2C1681844018952, suffix=, logDir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/WALs/jenkins-hbase4.apache.org,34053,1681844018952, archiveDir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/oldWALs, maxLogs=32 2023-04-18 18:53:40,442 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34053,1681844018952}] 2023-04-18 18:53:40,462 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38449,DS-d78a5de1-2028-488e-9841-5b8299adb3db,DISK] 2023-04-18 18:53:40,466 INFO [RS:0;jenkins-hbase4:34053] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/WALs/jenkins-hbase4.apache.org,34053,1681844018952/jenkins-hbase4.apache.org%2C34053%2C1681844018952.1681844020442 2023-04-18 18:53:40,466 DEBUG [RS:0;jenkins-hbase4:34053] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38449,DS-d78a5de1-2028-488e-9841-5b8299adb3db,DISK]] 2023-04-18 18:53:40,628 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:40,631 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-18 18:53:40,635 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51998, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-18 18:53:40,649 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-18 18:53:40,649 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-18 18:53:40,653 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34053%2C1681844018952.meta, suffix=.meta, logDir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/WALs/jenkins-hbase4.apache.org,34053,1681844018952, archiveDir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/oldWALs, maxLogs=32 2023-04-18 18:53:40,672 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38449,DS-d78a5de1-2028-488e-9841-5b8299adb3db,DISK] 2023-04-18 18:53:40,678 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/WALs/jenkins-hbase4.apache.org,34053,1681844018952/jenkins-hbase4.apache.org%2C34053%2C1681844018952.meta.1681844020656.meta 2023-04-18 18:53:40,678 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38449,DS-d78a5de1-2028-488e-9841-5b8299adb3db,DISK]] 2023-04-18 18:53:40,679 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-18 18:53:40,681 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-18 18:53:40,704 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-18 18:53:40,709 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-18 18:53:40,715 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-18 18:53:40,715 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 18:53:40,715 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-18 18:53:40,715 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-18 18:53:40,719 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-18 18:53:40,721 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/info 2023-04-18 18:53:40,721 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/info 2023-04-18 18:53:40,722 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-18 18:53:40,723 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:53:40,723 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-18 18:53:40,724 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/rep_barrier 2023-04-18 18:53:40,725 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/rep_barrier 2023-04-18 18:53:40,725 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-18 18:53:40,726 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:53:40,727 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-18 18:53:40,728 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/table 2023-04-18 18:53:40,728 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/table 2023-04-18 18:53:40,729 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-18 18:53:40,730 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:53:40,732 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740 2023-04-18 18:53:40,735 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740 2023-04-18 18:53:40,739 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 18:53:40,742 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-18 18:53:40,743 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66324148, jitterRate=-0.011693179607391357}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 18:53:40,744 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-18 18:53:40,756 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1681844020620 2023-04-18 18:53:40,774 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-18 18:53:40,775 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-18 18:53:40,776 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34053,1681844018952, state=OPEN 2023-04-18 18:53:40,779 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-18 18:53:40,780 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-18 18:53:40,785 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-18 18:53:40,786 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34053,1681844018952 in 337 msec 2023-04-18 18:53:40,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-18 18:53:40,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 541 msec 2023-04-18 18:53:40,800 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 800 msec 2023-04-18 18:53:40,800 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1681844020800, completionTime=-1 2023-04-18 18:53:40,801 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-18 18:53:40,801 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-18 18:53:40,881 DEBUG [hconnection-0x12cc5459-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:53:40,885 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52014, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:53:40,907 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-18 18:53:40,907 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681844080907 2023-04-18 18:53:40,908 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681844140908 2023-04-18 18:53:40,908 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 106 msec 2023-04-18 18:53:40,938 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39391,1681844017730-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-18 18:53:40,938 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39391,1681844017730-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-18 18:53:40,938 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39391,1681844017730-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-18 18:53:40,940 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:39391, period=300000, unit=MILLISECONDS is enabled. 2023-04-18 18:53:40,941 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-18 18:53:40,952 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-18 18:53:40,959 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-18 18:53:40,961 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-18 18:53:40,973 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-18 18:53:40,976 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-18 18:53:40,979 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-18 18:53:41,012 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/hbase/namespace/8ca45fd8bf2d7472adff2285bed4eeeb 2023-04-18 18:53:41,015 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/hbase/namespace/8ca45fd8bf2d7472adff2285bed4eeeb empty. 2023-04-18 18:53:41,017 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/hbase/namespace/8ca45fd8bf2d7472adff2285bed4eeeb 2023-04-18 18:53:41,017 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-18 18:53:41,460 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-18 18:53:41,463 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 8ca45fd8bf2d7472adff2285bed4eeeb, NAME => 'hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp 2023-04-18 18:53:41,878 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 18:53:41,879 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 8ca45fd8bf2d7472adff2285bed4eeeb, disabling compactions & flushes 2023-04-18 18:53:41,879 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb. 2023-04-18 18:53:41,879 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb. 2023-04-18 18:53:41,879 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb. after waiting 0 ms 2023-04-18 18:53:41,879 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb. 2023-04-18 18:53:41,879 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb. 2023-04-18 18:53:41,879 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 8ca45fd8bf2d7472adff2285bed4eeeb: 2023-04-18 18:53:41,885 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-18 18:53:41,903 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681844021888"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681844021888"}]},"ts":"1681844021888"} 2023-04-18 18:53:41,933 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-18 18:53:41,935 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-18 18:53:41,939 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844021935"}]},"ts":"1681844021935"} 2023-04-18 18:53:41,944 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-18 18:53:41,958 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=8ca45fd8bf2d7472adff2285bed4eeeb, ASSIGN}] 2023-04-18 18:53:41,961 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=8ca45fd8bf2d7472adff2285bed4eeeb, ASSIGN 2023-04-18 18:53:41,963 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=8ca45fd8bf2d7472adff2285bed4eeeb, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34053,1681844018952; forceNewPlan=false, retain=false 2023-04-18 18:53:42,114 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=8ca45fd8bf2d7472adff2285bed4eeeb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:42,114 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681844022114"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681844022114"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681844022114"}]},"ts":"1681844022114"} 2023-04-18 18:53:42,119 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 8ca45fd8bf2d7472adff2285bed4eeeb, server=jenkins-hbase4.apache.org,34053,1681844018952}] 2023-04-18 18:53:42,281 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb. 2023-04-18 18:53:42,282 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8ca45fd8bf2d7472adff2285bed4eeeb, NAME => 'hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb.', STARTKEY => '', ENDKEY => ''} 2023-04-18 18:53:42,283 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 8ca45fd8bf2d7472adff2285bed4eeeb 2023-04-18 18:53:42,283 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 18:53:42,283 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 8ca45fd8bf2d7472adff2285bed4eeeb 2023-04-18 18:53:42,283 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 8ca45fd8bf2d7472adff2285bed4eeeb 2023-04-18 18:53:42,286 INFO [StoreOpener-8ca45fd8bf2d7472adff2285bed4eeeb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 8ca45fd8bf2d7472adff2285bed4eeeb 2023-04-18 18:53:42,288 DEBUG [StoreOpener-8ca45fd8bf2d7472adff2285bed4eeeb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/namespace/8ca45fd8bf2d7472adff2285bed4eeeb/info 2023-04-18 18:53:42,288 DEBUG [StoreOpener-8ca45fd8bf2d7472adff2285bed4eeeb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/namespace/8ca45fd8bf2d7472adff2285bed4eeeb/info 2023-04-18 18:53:42,289 INFO [StoreOpener-8ca45fd8bf2d7472adff2285bed4eeeb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8ca45fd8bf2d7472adff2285bed4eeeb columnFamilyName info 2023-04-18 18:53:42,289 INFO [StoreOpener-8ca45fd8bf2d7472adff2285bed4eeeb-1] regionserver.HStore(310): Store=8ca45fd8bf2d7472adff2285bed4eeeb/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:53:42,291 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/namespace/8ca45fd8bf2d7472adff2285bed4eeeb 2023-04-18 18:53:42,292 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/namespace/8ca45fd8bf2d7472adff2285bed4eeeb 2023-04-18 18:53:42,296 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 8ca45fd8bf2d7472adff2285bed4eeeb 2023-04-18 18:53:42,300 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/namespace/8ca45fd8bf2d7472adff2285bed4eeeb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-18 18:53:42,301 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 8ca45fd8bf2d7472adff2285bed4eeeb; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71052504, jitterRate=0.05876481533050537}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-18 18:53:42,301 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 8ca45fd8bf2d7472adff2285bed4eeeb: 2023-04-18 18:53:42,303 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb., pid=6, masterSystemTime=1681844022273 2023-04-18 18:53:42,307 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb. 2023-04-18 18:53:42,307 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb. 2023-04-18 18:53:42,308 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=8ca45fd8bf2d7472adff2285bed4eeeb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:42,309 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681844022308"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681844022308"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681844022308"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681844022308"}]},"ts":"1681844022308"} 2023-04-18 18:53:42,316 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-18 18:53:42,317 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 8ca45fd8bf2d7472adff2285bed4eeeb, server=jenkins-hbase4.apache.org,34053,1681844018952 in 194 msec 2023-04-18 18:53:42,320 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-18 18:53:42,321 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=8ca45fd8bf2d7472adff2285bed4eeeb, ASSIGN in 358 msec 2023-04-18 18:53:42,322 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-18 18:53:42,322 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844022322"}]},"ts":"1681844022322"} 2023-04-18 18:53:42,326 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-18 18:53:42,334 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-18 18:53:42,337 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3720 sec 2023-04-18 18:53:42,377 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-18 18:53:42,379 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-18 18:53:42,379 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-18 18:53:42,425 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-18 18:53:42,443 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-18 18:53:42,451 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 35 msec 2023-04-18 18:53:42,461 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-18 18:53:42,479 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-18 18:53:42,486 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 24 msec 2023-04-18 18:53:42,499 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-18 18:53:42,502 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-18 18:53:42,504 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.460sec 2023-04-18 18:53:42,506 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-18 18:53:42,507 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-18 18:53:42,507 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-18 18:53:42,509 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39391,1681844017730-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-18 18:53:42,509 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39391,1681844017730-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-18 18:53:42,520 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-18 18:53:42,606 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x461317a6 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:53:42,612 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39efd3b0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:53:42,628 DEBUG [hconnection-0xb966214-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:53:42,642 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52028, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:53:42,653 INFO [Listener at localhost/36951] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,39391,1681844017730 2023-04-18 18:53:42,676 INFO [Listener at localhost/36951] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=209, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=310, ProcessCount=170, AvailableMemoryMB=7526 2023-04-18 18:53:42,693 DEBUG [Listener at localhost/36951] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-18 18:53:42,697 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33236, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-18 18:53:42,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-18 18:53:42,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-18 18:53:42,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-18 18:53:42,717 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-18 18:53:42,720 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-18 18:53:42,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-18 18:53:42,723 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:42,724 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e empty. 2023-04-18 18:53:42,726 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:42,727 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 18:53:42,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-18 18:53:42,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-18 18:53:43,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-18 18:53:43,147 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-18 18:53:43,150 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => d1a51bf079ac6b54cf5bff45ab0a5c8e, NAME => 'TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp 2023-04-18 18:53:43,166 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 18:53:43,166 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing d1a51bf079ac6b54cf5bff45ab0a5c8e, disabling compactions & flushes 2023-04-18 18:53:43,166 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:43,167 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:43,167 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. after waiting 0 ms 2023-04-18 18:53:43,167 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:43,167 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:43,167 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:43,171 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-18 18:53:43,173 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844023173"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681844023173"}]},"ts":"1681844023173"} 2023-04-18 18:53:43,176 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-18 18:53:43,178 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-18 18:53:43,178 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844023178"}]},"ts":"1681844023178"} 2023-04-18 18:53:43,180 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-18 18:53:43,184 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d1a51bf079ac6b54cf5bff45ab0a5c8e, ASSIGN}] 2023-04-18 18:53:43,186 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d1a51bf079ac6b54cf5bff45ab0a5c8e, ASSIGN 2023-04-18 18:53:43,188 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d1a51bf079ac6b54cf5bff45ab0a5c8e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34053,1681844018952; forceNewPlan=false, retain=false 2023-04-18 18:53:43,339 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=d1a51bf079ac6b54cf5bff45ab0a5c8e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:43,340 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844023339"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681844023339"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681844023339"}]},"ts":"1681844023339"} 2023-04-18 18:53:43,344 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952}] 2023-04-18 18:53:43,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-18 18:53:43,505 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:43,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d1a51bf079ac6b54cf5bff45ab0a5c8e, NAME => 'TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.', STARTKEY => '', ENDKEY => ''} 2023-04-18 18:53:43,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:43,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 18:53:43,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:43,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:43,509 INFO [StoreOpener-d1a51bf079ac6b54cf5bff45ab0a5c8e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:43,512 DEBUG [StoreOpener-d1a51bf079ac6b54cf5bff45ab0a5c8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A 2023-04-18 18:53:43,512 DEBUG [StoreOpener-d1a51bf079ac6b54cf5bff45ab0a5c8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A 2023-04-18 18:53:43,514 INFO [StoreOpener-d1a51bf079ac6b54cf5bff45ab0a5c8e-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:53:43,515 INFO [StoreOpener-d1a51bf079ac6b54cf5bff45ab0a5c8e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d1a51bf079ac6b54cf5bff45ab0a5c8e columnFamilyName A 2023-04-18 18:53:43,516 INFO [StoreOpener-d1a51bf079ac6b54cf5bff45ab0a5c8e-1] regionserver.HStore(310): Store=d1a51bf079ac6b54cf5bff45ab0a5c8e/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:53:43,516 INFO [StoreOpener-d1a51bf079ac6b54cf5bff45ab0a5c8e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:43,519 DEBUG [StoreOpener-d1a51bf079ac6b54cf5bff45ab0a5c8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B 2023-04-18 18:53:43,520 DEBUG [StoreOpener-d1a51bf079ac6b54cf5bff45ab0a5c8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B 2023-04-18 18:53:43,520 INFO [StoreOpener-d1a51bf079ac6b54cf5bff45ab0a5c8e-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:53:43,521 INFO [StoreOpener-d1a51bf079ac6b54cf5bff45ab0a5c8e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d1a51bf079ac6b54cf5bff45ab0a5c8e columnFamilyName B 2023-04-18 18:53:43,522 INFO [StoreOpener-d1a51bf079ac6b54cf5bff45ab0a5c8e-1] regionserver.HStore(310): Store=d1a51bf079ac6b54cf5bff45ab0a5c8e/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:53:43,522 INFO [StoreOpener-d1a51bf079ac6b54cf5bff45ab0a5c8e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:43,525 DEBUG [StoreOpener-d1a51bf079ac6b54cf5bff45ab0a5c8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C 2023-04-18 18:53:43,525 DEBUG [StoreOpener-d1a51bf079ac6b54cf5bff45ab0a5c8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C 2023-04-18 18:53:43,525 INFO [StoreOpener-d1a51bf079ac6b54cf5bff45ab0a5c8e-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:53:43,525 INFO [StoreOpener-d1a51bf079ac6b54cf5bff45ab0a5c8e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d1a51bf079ac6b54cf5bff45ab0a5c8e columnFamilyName C 2023-04-18 18:53:43,526 INFO [StoreOpener-d1a51bf079ac6b54cf5bff45ab0a5c8e-1] regionserver.HStore(310): Store=d1a51bf079ac6b54cf5bff45ab0a5c8e/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:53:43,527 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:43,530 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:43,531 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:43,534 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 18:53:43,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:43,541 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-18 18:53:43,542 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d1a51bf079ac6b54cf5bff45ab0a5c8e; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69225118, jitterRate=0.03153464198112488}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 18:53:43,542 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:43,543 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., pid=11, masterSystemTime=1681844023498 2023-04-18 18:53:43,546 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:43,546 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:43,547 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=d1a51bf079ac6b54cf5bff45ab0a5c8e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:43,547 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844023547"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681844023547"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681844023547"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681844023547"}]},"ts":"1681844023547"} 2023-04-18 18:53:43,553 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-18 18:53:43,554 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 in 207 msec 2023-04-18 18:53:43,557 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-18 18:53:43,558 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d1a51bf079ac6b54cf5bff45ab0a5c8e, ASSIGN in 370 msec 2023-04-18 18:53:43,559 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-18 18:53:43,559 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844023559"}]},"ts":"1681844023559"} 2023-04-18 18:53:43,561 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-18 18:53:43,565 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-18 18:53:43,568 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 853 msec 2023-04-18 18:53:43,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-18 18:53:43,847 INFO [Listener at localhost/36951] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-18 18:53:43,852 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x5d4f988c to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:53:43,858 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23162110, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:53:43,860 DEBUG [Listener at localhost/36951] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:53:43,866 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33402, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:53:43,869 DEBUG [Listener at localhost/36951] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-18 18:53:43,871 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51502, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-18 18:53:43,879 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x72ea4a65 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:53:43,884 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e82daee, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:53:43,884 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x1645b6b0 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:53:43,890 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e8196e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:53:43,890 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x7b52aa42 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:53:43,894 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2e2005d0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:53:43,895 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x1710407d to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:53:43,900 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57de6c0d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:53:43,900 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x7bc6d729 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:53:43,905 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@75f5a5e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:53:43,907 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x00375c9d to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:53:43,918 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@64804459, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:53:43,919 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x2ec2fae8 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:53:43,955 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@142b9819, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:53:43,956 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x27685d29 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:53:43,961 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fcbe28d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:53:43,962 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x5979620a to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:53:43,966 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c9ded19, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:53:44,011 DEBUG [hconnection-0x74c2cd1b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:53:44,014 DEBUG [hconnection-0x6ff4c9b4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:53:44,014 DEBUG [hconnection-0x7576e4ed-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:53:44,015 DEBUG [hconnection-0x55d4bc5a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:53:44,014 DEBUG [hconnection-0x62a93ad6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:53:44,014 DEBUG [hconnection-0x2a27259-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:53:44,014 DEBUG [hconnection-0x156ee019-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:53:44,017 DEBUG [hconnection-0x7d721dea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:53:44,026 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33426, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:53:44,028 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33410, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:53:44,028 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33436, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:53:44,032 DEBUG [hconnection-0x1531940c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:53:44,033 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33446, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:53:44,034 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33454, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:53:44,054 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33462, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:53:44,055 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33476, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:53:44,056 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33468, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:53:44,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:44,069 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33490, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:53:44,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:44,091 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:44,091 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:44,092 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:44,092 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:44,099 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:44,099 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:44,101 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:44,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:44,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:44,103 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:44,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:44,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:44,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:44,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:44,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:44,110 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:44,110 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:44,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:44,124 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:44,124 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:44,128 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:44,129 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:44,129 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:44,132 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:44,137 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:44,141 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 18:53:44,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:44,145 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:44,155 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:44,156 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:44,157 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:44,157 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:44,157 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:44,157 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:44,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:44,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844084248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:44,298 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:44,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844084249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:44,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844024214, max timestamp=1681844024248Num uniques 859; 2023-04-18 18:53:44,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:53:44,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681844024214, max timestamp=1681844024248Num uniques 859; 2023-04-18 18:53:44,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-18 18:53:44,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681844024214, max timestamp=1681844024263Num uniques 859; 2023-04-18 18:53:44,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-18 18:53:44,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:44,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844084387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:44,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:44,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844084397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:44,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681844024377, max timestamp=1681844024391Num uniques 1009; 2023-04-18 18:53:44,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-18 18:53:44,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681844024377, max timestamp=1681844024391Num uniques 1018; 2023-04-18 18:53:44,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-18 18:53:44,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681844024377, max timestamp=1681844024391Num uniques 1009; 2023-04-18 18:53:44,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-18 18:53:44,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:44,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844084509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:44,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:44,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844084509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:44,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681844024498, max timestamp=1681844024510Num uniques 1027; 2023-04-18 18:53:44,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 18:53:44,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681844024498, max timestamp=1681844024510Num uniques 1027; 2023-04-18 18:53:44,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 18:53:44,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681844024498, max timestamp=1681844024510Num uniques 1027; 2023-04-18 18:53:44,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 18:53:44,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:44,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844084587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:44,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:44,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:44,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844084589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:44,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844084585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:44,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:44,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844084628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:44,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:44,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844084635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:44,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:44,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844084695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:44,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:44,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844084700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:44,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:44,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844084701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:44,744 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/b7e9409cfd0d49b083546649cb4337f2 2023-04-18 18:53:44,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:44,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844084835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:44,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:44,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844084842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:44,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:44,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844084903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:44,906 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/64315ef840d64b1bb93f1b38b59710ac 2023-04-18 18:53:44,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:44,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:44,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844084915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:44,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844084915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:44,997 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/4d77308ebb3048b694f07a26e7cbf651 2023-04-18 18:53:45,011 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/b7e9409cfd0d49b083546649cb4337f2 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/b7e9409cfd0d49b083546649cb4337f2 2023-04-18 18:53:45,023 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/b7e9409cfd0d49b083546649cb4337f2, entries=100, sequenceid=11, filesize=9.3 K 2023-04-18 18:53:45,033 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/64315ef840d64b1bb93f1b38b59710ac as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/64315ef840d64b1bb93f1b38b59710ac 2023-04-18 18:53:45,053 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/64315ef840d64b1bb93f1b38b59710ac, entries=100, sequenceid=11, filesize=9.3 K 2023-04-18 18:53:45,057 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/4d77308ebb3048b694f07a26e7cbf651 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/4d77308ebb3048b694f07a26e7cbf651 2023-04-18 18:53:45,076 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/4d77308ebb3048b694f07a26e7cbf651, entries=100, sequenceid=11, filesize=9.3 K 2023-04-18 18:53:45,086 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 944ms, sequenceid=11, compaction requested=false 2023-04-18 18:53:45,089 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-18 18:53:45,092 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:45,092 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:45,092 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:45,093 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:45,093 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,094 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:45,094 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:45,097 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:45,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:45,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:45,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:45,099 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:45,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:45,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,101 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:45,101 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:45,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:45,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:45,102 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:45,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a0dcd9f[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:45,102 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:45,106 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:45,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:45,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:45,106 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:45,106 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:45,107 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:45,108 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,108 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:45,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:45,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:45,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:45,116 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:45,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:45,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:45,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:45,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:45,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:45,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:45,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:45,122 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:45,123 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:45,123 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:45,123 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:45,123 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:45,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:45,127 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:45,127 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:45,127 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:45,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:45,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:45,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:45,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:45,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:45,129 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:45,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:45,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:45,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,135 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,155 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:45,156 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:45,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:45,156 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:45,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:45,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:45,157 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,157 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:45,157 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:45,157 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,157 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:45,158 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:45,158 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:45,159 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:45,159 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:45,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:45,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:45,170 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=147.60 KB heapSize=423.84 KB 2023-04-18 18:53:45,176 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:45,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 18:53:45,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:45,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 18:53:45,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:45,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 18:53:45,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844085225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844085232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844085227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,258 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/c73adec37e4b416cb5d6bbb61efe1ddb 2023-04-18 18:53:45,259 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:45,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:45,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:45,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844085257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844085257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:45,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:45,308 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:45,309 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:45,313 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:45,313 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:45,314 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:45,314 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:45,316 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:45,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:45,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:45,316 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:45,317 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,317 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:45,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:45,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:45,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:45,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:45,319 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:45,319 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:45,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:45,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:45,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:45,321 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:45,321 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:45,321 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:45,321 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:45,325 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:45,325 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:45,325 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,325 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:45,325 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:45,334 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:45,334 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:45,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:45,336 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:45,336 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:45,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:45,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:45,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:45,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:45,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74798dd0[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:45,368 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:45,368 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:45,369 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:45,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:45,370 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:45,370 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:45,370 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:45,371 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,371 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:45,387 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:45,387 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:45,388 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:45,388 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:45,390 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/a81c73b161464bbcaf9e583bc709ade0 2023-04-18 18:53:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:45,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:45,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844085367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:45,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:45,395 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:45,395 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:45,395 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:45,395 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:45,395 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:45,400 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:45,400 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:45,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:45,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:45,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:45,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:45,400 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:45,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:45,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:45,403 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:45,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:45,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,406 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:45,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844085389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844085389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844085391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844085397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,421 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:45,421 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:45,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:45,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:45,422 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,422 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:45,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:45,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:45,422 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:45,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:45,422 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:45,423 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:45,423 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:45,423 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,423 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:45,423 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:45,423 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:45,468 WARN [DataStreamer for file /user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/3d87554627f141e093bcaf68d5bdd9c5] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-18 18:53:45,470 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/3d87554627f141e093bcaf68d5bdd9c5 2023-04-18 18:53:45,484 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/c73adec37e4b416cb5d6bbb61efe1ddb as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/c73adec37e4b416cb5d6bbb61efe1ddb 2023-04-18 18:53:45,499 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/c73adec37e4b416cb5d6bbb61efe1ddb, entries=150, sequenceid=84, filesize=11.6 K 2023-04-18 18:53:45,501 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/a81c73b161464bbcaf9e583bc709ade0 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/a81c73b161464bbcaf9e583bc709ade0 2023-04-18 18:53:45,511 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/a81c73b161464bbcaf9e583bc709ade0, entries=150, sequenceid=84, filesize=11.6 K 2023-04-18 18:53:45,521 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/3d87554627f141e093bcaf68d5bdd9c5 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/3d87554627f141e093bcaf68d5bdd9c5 2023-04-18 18:53:45,523 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:45,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:45,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:45,534 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/3d87554627f141e093bcaf68d5bdd9c5, entries=150, sequenceid=84, filesize=11.6 K 2023-04-18 18:53:45,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:45,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:45,539 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:45,539 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:45,540 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:45,541 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:45,541 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:45,541 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:45,543 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,543 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:45,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:45,543 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,543 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:45,544 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~423.80 KB/433968, currentSize=40.25 KB/41220 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 375ms, sequenceid=84, compaction requested=false 2023-04-18 18:53:45,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:45,545 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:45,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:45,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:45,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:45,548 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:45,548 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:45,551 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:45,552 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:45,552 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:45,553 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:45,553 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:45,553 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-18 18:53:45,554 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:45,554 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:45,554 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:45,554 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:45,554 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:45,554 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:45,603 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:45,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:45,651 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/f84fe57054da426698203ecffa3f0597 2023-04-18 18:53:45,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844085681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844085681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844085686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844085687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844085688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681844025675, max timestamp=1681844025678Num uniques 859; 2023-04-18 18:53:45,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-18 18:53:45,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681844025675, max timestamp=1681844025678Num uniques 859; 2023-04-18 18:53:45,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-18 18:53:45,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844085816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844085820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844085823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844085824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844085824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844085926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844085943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,955 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844085943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,955 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844085944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:45,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:45,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844085944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,120 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/65cb3f84cdd74fcfa65e6a31392cba2c 2023-04-18 18:53:46,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:46,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844086146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:46,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844086160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:46,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844086160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:46,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844086179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:46,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844086160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,240 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/aaa88160a4fa416492fed07aabeba119 2023-04-18 18:53:46,261 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/f84fe57054da426698203ecffa3f0597 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/f84fe57054da426698203ecffa3f0597 2023-04-18 18:53:46,275 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/f84fe57054da426698203ecffa3f0597, entries=150, sequenceid=93, filesize=11.6 K 2023-04-18 18:53:46,277 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/65cb3f84cdd74fcfa65e6a31392cba2c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/65cb3f84cdd74fcfa65e6a31392cba2c 2023-04-18 18:53:46,303 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-18 18:53:46,308 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/65cb3f84cdd74fcfa65e6a31392cba2c, entries=150, sequenceid=93, filesize=11.6 K 2023-04-18 18:53:46,313 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/aaa88160a4fa416492fed07aabeba119 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/aaa88160a4fa416492fed07aabeba119 2023-04-18 18:53:46,342 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/aaa88160a4fa416492fed07aabeba119, entries=150, sequenceid=93, filesize=11.6 K 2023-04-18 18:53:46,345 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=151.28 KB/154910 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 792ms, sequenceid=93, compaction requested=true 2023-04-18 18:53:46,345 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:46,345 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:46,345 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:46,358 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:46,359 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:46,359 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:46,364 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:46,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:46,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:46,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:46,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@583fb4c7[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:46,370 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:46,370 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:46,371 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:46,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:46,371 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:46,371 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:46,372 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:46,372 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,372 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:46,381 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:46,381 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:46,381 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:46,381 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:46,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:46,393 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:46,393 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:46,393 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:46,393 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:46,393 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:46,397 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:46,397 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:46,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:46,397 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:46,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:46,398 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:46,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:46,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,400 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,407 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:46,407 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:46,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:46,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:46,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:46,407 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:46,408 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:46,408 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,408 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:46,408 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:46,408 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:46,407 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:46,407 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:46,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:46,413 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:46,413 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:46,464 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-18 18:53:46,475 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-18 18:53:46,488 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=157.99 KB heapSize=461.17 KB 2023-04-18 18:53:46,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:46,489 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:46,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:53:46,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:46,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:53:46,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:46,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:53:46,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:46,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844086489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:46,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:46,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844086490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844086493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:46,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844086496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:46,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844086496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,513 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:46,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:46,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:46,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:46,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:46,552 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:46,552 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:46,560 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:46,560 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:46,561 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:46,562 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:46,567 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:46,573 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:46,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:46,574 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,574 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:46,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:46,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:46,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:46,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:46,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:46,585 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:46,585 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:46,586 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:46,593 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:46,593 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:46,593 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:46,593 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:46,593 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:46,594 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:46,596 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:46,596 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:46,596 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,596 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:46,596 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:46,599 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:46,601 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/1d16005b3db84664b77910e34fd407e8 2023-04-18 18:53:46,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:46,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844086600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,604 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:46,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:46,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:46,606 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:46,606 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:46,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:46,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:46,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:46,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844086600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:46,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844086600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:46,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844086604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:46,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:46,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@581e9e89[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:46,612 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:46,612 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:46,621 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:46,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:46,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:46,622 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:46,622 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:46,622 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:46,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:46,634 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,634 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:46,635 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:46,635 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:46,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:46,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:46,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:46,648 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:46,648 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:46,648 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:46,648 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:46,648 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:46,650 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:46,650 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:46,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:46,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:46,650 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:46,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:46,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:46,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:46,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:46,665 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,668 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/862fbf81283d4c70ae6de7d87951bf89 2023-04-18 18:53:46,668 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,682 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:46,682 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:46,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:46,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:46,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:46,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:46,684 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,684 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:46,684 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:46,684 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:46,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:46,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:46,684 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,684 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:46,684 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:46,684 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:46,685 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:46,685 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:46,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:46,744 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=76.04 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/b01a6603515e4d4d949c66c67cece9f5 2023-04-18 18:53:46,756 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/1d16005b3db84664b77910e34fd407e8 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/1d16005b3db84664b77910e34fd407e8 2023-04-18 18:53:46,774 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/1d16005b3db84664b77910e34fd407e8, entries=250, sequenceid=130, filesize=16.3 K 2023-04-18 18:53:46,779 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/862fbf81283d4c70ae6de7d87951bf89 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/862fbf81283d4c70ae6de7d87951bf89 2023-04-18 18:53:46,785 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:46,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:46,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:46,791 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/862fbf81283d4c70ae6de7d87951bf89, entries=150, sequenceid=130, filesize=11.7 K 2023-04-18 18:53:46,793 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/b01a6603515e4d4d949c66c67cece9f5 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/b01a6603515e4d4d949c66c67cece9f5 2023-04-18 18:53:46,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:46,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844086810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:46,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:46,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:46,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:46,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844086813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:46,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844086812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,826 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:46,827 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:46,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844086812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,833 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/b01a6603515e4d4d949c66c67cece9f5, entries=150, sequenceid=130, filesize=11.7 K 2023-04-18 18:53:46,833 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:46,834 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:46,835 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:46,835 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:46,835 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~164.70 KB/168650, heapSize ~481.05 KB/492592, currentSize=13.42 KB/13740 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 347ms, sequenceid=130, compaction requested=true 2023-04-18 18:53:46,835 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:46,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:46,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:46,839 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,839 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:46,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:46,840 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:53:46,840 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:46,840 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:46,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:46,841 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:53:46,840 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:53:46,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:46,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:46,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:46,841 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:46,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:46,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:46,842 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:46,842 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:46,843 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:46,843 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:46,843 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:46,843 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:46,843 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:46,844 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-18 18:53:46,844 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:46,844 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:46,844 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:46,844 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:46,844 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:46,844 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:46,852 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49980 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:53:46,852 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45250 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:53:46,855 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/B is initiating minor compaction (all files) 2023-04-18 18:53:46,855 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/B in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:46,855 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/64315ef840d64b1bb93f1b38b59710ac, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/a81c73b161464bbcaf9e583bc709ade0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/65cb3f84cdd74fcfa65e6a31392cba2c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/862fbf81283d4c70ae6de7d87951bf89] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=44.2 K 2023-04-18 18:53:46,856 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/A is initiating minor compaction (all files) 2023-04-18 18:53:46,856 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/A in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:46,856 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/b7e9409cfd0d49b083546649cb4337f2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/c73adec37e4b416cb5d6bbb61efe1ddb, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/f84fe57054da426698203ecffa3f0597, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/1d16005b3db84664b77910e34fd407e8] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=48.8 K 2023-04-18 18:53:46,857 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 64315ef840d64b1bb93f1b38b59710ac, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681844024122 2023-04-18 18:53:46,861 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting b7e9409cfd0d49b083546649cb4337f2, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681844024122 2023-04-18 18:53:46,865 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting a81c73b161464bbcaf9e583bc709ade0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1681844024582 2023-04-18 18:53:46,865 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting c73adec37e4b416cb5d6bbb61efe1ddb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1681844024582 2023-04-18 18:53:46,866 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting f84fe57054da426698203ecffa3f0597, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681844025216 2023-04-18 18:53:46,868 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 65cb3f84cdd74fcfa65e6a31392cba2c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681844025216 2023-04-18 18:53:46,868 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 862fbf81283d4c70ae6de7d87951bf89, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1681844025819 2023-04-18 18:53:46,868 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 1d16005b3db84664b77910e34fd407e8, keycount=250, bloomtype=ROW, size=16.3 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1681844025819 2023-04-18 18:53:46,907 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/f7f96e3216b14b60b159ee0369fbdb64 2023-04-18 18:53:46,928 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#A#compaction#13 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:46,947 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#B#compaction#15 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:47,012 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/189ac20ad72a4873a739d175b6e85809 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/189ac20ad72a4873a739d175b6e85809 2023-04-18 18:53:47,030 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:47,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:47,047 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/B of d1a51bf079ac6b54cf5bff45ab0a5c8e into 189ac20ad72a4873a739d175b6e85809(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:47,047 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:47,048 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/B, priority=12, startTime=1681844026840; duration=0sec 2023-04-18 18:53:47,049 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:53:47,049 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:53:47,055 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45250 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:53:47,055 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/C is initiating minor compaction (all files) 2023-04-18 18:53:47,055 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/C in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:47,055 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/4d77308ebb3048b694f07a26e7cbf651, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/3d87554627f141e093bcaf68d5bdd9c5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/aaa88160a4fa416492fed07aabeba119, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/b01a6603515e4d4d949c66c67cece9f5] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=44.2 K 2023-04-18 18:53:47,056 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 4d77308ebb3048b694f07a26e7cbf651, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681844024122 2023-04-18 18:53:47,057 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 3d87554627f141e093bcaf68d5bdd9c5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1681844024582 2023-04-18 18:53:47,058 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting aaa88160a4fa416492fed07aabeba119, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681844025216 2023-04-18 18:53:47,059 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b01a6603515e4d4d949c66c67cece9f5, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1681844025819 2023-04-18 18:53:47,082 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#C#compaction#16 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:47,125 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/fbd61286b90946398d594e64bb17d43f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/fbd61286b90946398d594e64bb17d43f 2023-04-18 18:53:47,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844027078, max timestamp=1681844027134Num uniques 859; 2023-04-18 18:53:47,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:53:47,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844027078, max timestamp=1681844027134Num uniques 859; 2023-04-18 18:53:47,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:53:47,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681844027116, max timestamp=1681844027134Num uniques 859; 2023-04-18 18:53:47,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-18 18:53:47,150 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/C of d1a51bf079ac6b54cf5bff45ab0a5c8e into fbd61286b90946398d594e64bb17d43f(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:47,150 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:47,150 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/C, priority=12, startTime=1681844026841; duration=0sec 2023-04-18 18:53:47,151 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:47,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681844027169, max timestamp=1681844027182Num uniques 1068; 2023-04-18 18:53:47,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-18 18:53:47,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681844027169, max timestamp=1681844027183Num uniques 1009; 2023-04-18 18:53:47,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681844027169, max timestamp=1681844027183Num uniques 1068; 2023-04-18 18:53:47,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-18 18:53:47,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-18 18:53:47,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681844027208, max timestamp=1681844027216Num uniques 1027; 2023-04-18 18:53:47,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-18 18:53:47,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681844027210, max timestamp=1681844027216Num uniques 1027; 2023-04-18 18:53:47,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 18:53:47,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681844027210, max timestamp=1681844027216Num uniques 1027; 2023-04-18 18:53:47,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 18:53:47,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681844027233, max timestamp=1681844027272Num uniques 1136; 2023-04-18 18:53:47,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-18 18:53:47,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681844027233, max timestamp=1681844027270Num uniques 1036; 2023-04-18 18:53:47,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681844027233, max timestamp=1681844027270Num uniques 1036; 2023-04-18 18:53:47,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-18 18:53:47,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-18 18:53:47,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681844027294, max timestamp=1681844027324Num uniques 1045; 2023-04-18 18:53:47,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-18 18:53:47,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681844027294, max timestamp=1681844027324Num uniques 1045; 2023-04-18 18:53:47,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-18 18:53:47,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681844027294, max timestamp=1681844027327Num uniques 1045; 2023-04-18 18:53:47,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-18 18:53:47,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681844027353, max timestamp=1681844027379Num uniques 1104; 2023-04-18 18:53:47,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-18 18:53:47,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681844027353, max timestamp=1681844027379Num uniques 1104; 2023-04-18 18:53:47,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-18 18:53:47,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681844027353, max timestamp=1681844027379Num uniques 1104; 2023-04-18 18:53:47,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-18 18:53:47,395 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/42474ee0cd554a808f06705dd745d4fb 2023-04-18 18:53:47,407 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/da06836ad8584a91a09d047350251a91 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/da06836ad8584a91a09d047350251a91 2023-04-18 18:53:47,439 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/A of d1a51bf079ac6b54cf5bff45ab0a5c8e into da06836ad8584a91a09d047350251a91(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:47,440 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:47,440 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/A, priority=12, startTime=1681844026838; duration=0sec 2023-04-18 18:53:47,440 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:47,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681844027423, max timestamp=1681844027465Num uniques 1013; 2023-04-18 18:53:47,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-18 18:53:47,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681844027423, max timestamp=1681844027465Num uniques 1013; 2023-04-18 18:53:47,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-18 18:53:47,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681844027423, max timestamp=1681844027465Num uniques 1063; 2023-04-18 18:53:47,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-18 18:53:47,482 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/910a494e285645e9aa67a8a70e2c37d2 2023-04-18 18:53:47,502 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/f7f96e3216b14b60b159ee0369fbdb64 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/f7f96e3216b14b60b159ee0369fbdb64 2023-04-18 18:53:47,521 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/f7f96e3216b14b60b159ee0369fbdb64, entries=50, sequenceid=135, filesize=7.1 K 2023-04-18 18:53:47,526 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/42474ee0cd554a808f06705dd745d4fb as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/42474ee0cd554a808f06705dd745d4fb 2023-04-18 18:53:47,549 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/42474ee0cd554a808f06705dd745d4fb, entries=50, sequenceid=135, filesize=7.1 K 2023-04-18 18:53:47,559 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/910a494e285645e9aa67a8a70e2c37d2 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/910a494e285645e9aa67a8a70e2c37d2 2023-04-18 18:53:47,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681844027506, max timestamp=1681844027563Num uniques 1072; 2023-04-18 18:53:47,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-04-18 18:53:47,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681844027506, max timestamp=1681844027563Num uniques 1072; 2023-04-18 18:53:47,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-04-18 18:53:47,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681844027506, max timestamp=1681844027563Num uniques 1072; 2023-04-18 18:53:47,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-04-18 18:53:47,594 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/910a494e285645e9aa67a8a70e2c37d2, entries=50, sequenceid=135, filesize=7.1 K 2023-04-18 18:53:47,599 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=87.22 KB/89310 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 756ms, sequenceid=135, compaction requested=false 2023-04-18 18:53:47,599 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:47,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:47,599 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:47,599 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=93.93 KB heapSize=254.86 KB 2023-04-18 18:53:47,599 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:47,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:47,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:47,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:47,601 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:47,601 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:53:47,601 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:47,601 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:53:47,601 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:47,601 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:53:47,609 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:47,609 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:47,610 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:47,610 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:47,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:47,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:47,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cdbd120[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:47,613 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:47,613 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:47,614 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:47,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:47,614 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:47,614 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:47,614 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:47,617 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:47,617 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:47,617 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:47,618 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,618 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:47,618 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:47,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:47,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:47,638 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:47,638 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:47,638 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:47,638 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:47,638 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:47,643 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:47,643 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:47,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:47,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:47,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:47,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:47,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:47,644 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:47,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:47,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:47,645 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:47,647 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:47,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:47,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844087643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,661 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,661 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:47,661 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:47,662 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:47,662 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,662 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:47,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:47,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:47,662 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:47,663 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:47,663 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:47,662 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:47,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:47,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:47,667 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:47,667 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:47,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:47,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844087648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:47,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:47,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:47,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844087646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:47,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844087658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:47,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844087666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:47,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844087765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,767 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:47,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:47,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:47,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:47,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844087772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:47,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:47,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844087772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844087773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:47,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844087774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:47,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:47,787 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:47,787 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:47,789 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:47,789 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:47,790 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:47,790 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:47,791 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:47,791 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:47,792 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,792 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:47,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:47,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:47,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:47,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:47,793 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:47,793 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:47,794 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:47,795 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:47,795 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:47,795 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:47,795 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:47,795 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:47,795 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:47,795 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:47,795 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:47,795 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,795 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:47,795 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:47,804 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:47,804 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:47,805 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:47,805 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:47,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:47,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:47,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:47,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:47,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44f4c466[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:47,808 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:47,808 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:47,810 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:47,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:47,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:47,810 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:47,810 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:47,810 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:47,810 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,810 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:47,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:47,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:47,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:47,814 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:47,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:47,822 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:47,822 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:47,822 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:47,823 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:47,823 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:47,825 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:47,825 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:47,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:47,825 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:47,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:47,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:47,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:47,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:47,828 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,831 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,855 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,855 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:47,855 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:47,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:47,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:47,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:47,857 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:47,857 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:47,855 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:47,855 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:47,858 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:47,858 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,858 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:47,858 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:47,858 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:47,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:47,888 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-18 18:53:47,889 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-18 18:53:47,958 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:47,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:47,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:47,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:47,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844087970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:47,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844087981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:47,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844087982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:47,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844087982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:47,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:47,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844087981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:48,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:48,002 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:48,002 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:48,003 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:48,003 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:48,006 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,006 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,009 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,009 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:48,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:48,010 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,010 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:48,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:48,016 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:48,017 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:48,017 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:48,025 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:48,025 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:48,025 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:48,025 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:48,025 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:48,026 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:48,026 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:48,026 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:48,026 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:48,026 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,026 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:48,026 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:48,029 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:48,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:48,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61c8d9b3[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:48,032 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:48,032 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,034 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,034 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:48,034 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:48,034 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:48,034 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,034 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:48,036 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,036 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:48,036 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:48,037 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:48,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:48,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:48,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:48,044 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:48,044 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:48,044 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:48,044 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:48,044 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:48,047 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,047 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:48,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,047 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:48,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,048 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:48,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:48,051 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,055 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:48,055 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,056 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,056 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,056 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,056 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,056 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,056 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,056 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:48,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:48,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:48,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:48,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:48,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:48,056 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:48,058 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:48,059 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:48,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:48,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:48,072 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/598de5b3676a483c8f030b4aea18d444 2023-04-18 18:53:48,144 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/f6130a44d1fc473db0271a112631bebd 2023-04-18 18:53:48,158 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:48,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:48,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:48,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:48,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:48,198 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:48,198 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:48,217 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:48,217 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:48,218 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,218 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,219 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,219 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:48,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:48,220 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,220 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:48,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:48,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:48,223 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:48,224 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:48,241 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:48,241 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:48,242 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:48,242 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:48,242 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:48,242 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:48,242 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,245 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:48,245 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:48,251 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,251 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:48,252 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,252 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:48,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:48,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:48,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:48,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:48,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d0ef9d5[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:48,271 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:48,271 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,274 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:48,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844088273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,274 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:48,274 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:48,274 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:48,275 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,275 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:48,276 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:48,277 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:48,277 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:48,277 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:48,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:48,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:48,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:48,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:48,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:48,285 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:48,285 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:48,285 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:48,285 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:48,285 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:48,288 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,288 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:48,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:48,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:48,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,290 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:48,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:48,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:48,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844088287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,293 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:48,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:48,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844088289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,295 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:48,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844088290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:48,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:48,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844088293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,305 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:48,306 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:48,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:48,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:48,307 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,307 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,307 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,307 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,307 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,307 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,307 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,307 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:48,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:48,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:48,308 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:48,308 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:48,410 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:48,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:48,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:48,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:48,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:48,425 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:48,425 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:48,430 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:48,430 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:48,431 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,431 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,432 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:48,432 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:48,433 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,433 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:48,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:48,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:48,436 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:48,436 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:48,437 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:48,437 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:48,437 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:48,437 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:48,438 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:48,438 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:48,438 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:48,438 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:48,438 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:48,438 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,438 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:48,438 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:48,440 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,440 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,440 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:48,443 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:48,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:48,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:48,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:48,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:48,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ffe6385[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:48,448 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:48,448 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,450 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,450 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:48,450 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:48,450 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:48,450 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,450 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:48,454 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:48,459 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:48,459 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:48,459 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:48,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:48,479 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:48,479 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:48,479 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:48,479 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:48,479 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:48,484 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,484 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:48,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,485 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:48,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,486 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:48,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,488 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,492 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:48,492 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,492 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:48,492 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:48,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:48,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:48,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:48,492 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,492 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:48,492 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,497 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,497 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,497 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,497 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:48,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:48,497 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:48,597 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:48,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:48,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:48,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:48,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:48,604 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:48,605 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:48,606 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:48,606 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:48,606 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,606 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,608 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:48,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:48,608 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,609 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,609 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:48,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,610 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:48,611 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:48,611 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:48,611 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:48,611 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:48,614 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:48,614 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:48,614 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:48,614 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:48,614 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:48,614 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:48,614 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:48,615 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:48,615 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:48,615 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,615 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:48,615 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:48,617 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,617 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:48,618 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,618 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:48,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:48,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dcb4de8[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:48,620 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:48,620 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,622 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,622 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:48,622 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:48,622 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:48,625 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:48,625 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:48,625 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:48,628 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,630 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:48,630 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:48,630 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:48,630 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:48,630 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:48,630 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:48,630 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:48,632 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,632 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:48,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,632 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:48,635 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,637 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,644 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:48,644 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:48,644 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:48,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:48,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:48,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:48,645 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:48,645 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,645 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:48,645 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,645 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:48,645 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,645 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,645 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,646 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,664 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/1f6aac2d31204612899b019afcfd2b5c 2023-04-18 18:53:48,678 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/598de5b3676a483c8f030b4aea18d444 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/598de5b3676a483c8f030b4aea18d444 2023-04-18 18:53:48,692 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/598de5b3676a483c8f030b4aea18d444, entries=300, sequenceid=288, filesize=19.0 K 2023-04-18 18:53:48,699 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/f6130a44d1fc473db0271a112631bebd as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/f6130a44d1fc473db0271a112631bebd 2023-04-18 18:53:48,714 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/f6130a44d1fc473db0271a112631bebd, entries=150, sequenceid=288, filesize=11.9 K 2023-04-18 18:53:48,715 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/1f6aac2d31204612899b019afcfd2b5c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/1f6aac2d31204612899b019afcfd2b5c 2023-04-18 18:53:48,734 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/1f6aac2d31204612899b019afcfd2b5c, entries=150, sequenceid=288, filesize=11.9 K 2023-04-18 18:53:48,737 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~274.73 KB/281328, currentSize=87.22 KB/89310 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 1138ms, sequenceid=288, compaction requested=true 2023-04-18 18:53:48,738 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:48,738 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:53:48,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:48,739 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:53:48,739 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:48,739 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:53:48,741 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38830 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:53:48,741 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/A is initiating minor compaction (all files) 2023-04-18 18:53:48,742 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/A in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:48,742 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/da06836ad8584a91a09d047350251a91, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/f7f96e3216b14b60b159ee0369fbdb64, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/598de5b3676a483c8f030b4aea18d444] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=37.9 K 2023-04-18 18:53:48,742 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 31510 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:53:48,742 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/B is initiating minor compaction (all files) 2023-04-18 18:53:48,742 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/B in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:48,743 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/189ac20ad72a4873a739d175b6e85809, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/42474ee0cd554a808f06705dd745d4fb, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/f6130a44d1fc473db0271a112631bebd] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=30.8 K 2023-04-18 18:53:48,744 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting da06836ad8584a91a09d047350251a91, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1681844025819 2023-04-18 18:53:48,745 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:48,747 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 189ac20ad72a4873a739d175b6e85809, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1681844025819 2023-04-18 18:53:48,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:48,747 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 42474ee0cd554a808f06705dd745d4fb, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1681844026492 2023-04-18 18:53:48,748 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting f7f96e3216b14b60b159ee0369fbdb64, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1681844026492 2023-04-18 18:53:48,748 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting f6130a44d1fc473db0271a112631bebd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681844027560 2023-04-18 18:53:48,749 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 598de5b3676a483c8f030b4aea18d444, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681844027560 2023-04-18 18:53:48,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:48,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:48,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:48,766 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:48,766 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:48,768 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:48,769 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:48,770 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,770 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,772 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:48,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:48,772 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:48,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:48,774 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,774 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:48,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:48,789 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-18 18:53:48,790 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:48,790 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:48,790 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:48,790 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:48,790 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:48,790 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:48,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:48,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:48,807 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:48,807 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:48,809 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#A#compaction#21 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:48,839 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:48,839 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:48,839 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:48,839 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:48,839 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:48,840 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:48,840 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:48,840 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:48,840 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:48,840 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,840 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:48,840 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:48,842 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,842 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:48,843 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,843 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:48,844 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#B#compaction#23 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:48,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:48,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a6918a3[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:48,852 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:48,853 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,854 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,855 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:48,857 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,858 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:48,858 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:48,858 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:48,861 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,861 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:48,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:48,861 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:48,862 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:48,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:48,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:48,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:48,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:48,879 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:48,879 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:48,879 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:48,879 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:48,880 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:48,882 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,882 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:48,882 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,884 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:48,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:48,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844088844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:48,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844088861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,896 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:48,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:48,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:48,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844088894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:48,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844088894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,899 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,899 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,899 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:48,899 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:48,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:48,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:48,899 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:48,900 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:48,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:48,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:48,899 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,900 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,900 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:48,900 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:48,900 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:48,900 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:48,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:48,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:48,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:48,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844088915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:48,982 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/d17096b41b774bec8310286fceb4bd9a 2023-04-18 18:53:49,000 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:49,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:49,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844088993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:49,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:49,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:49,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844088997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:49,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:49,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844089001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844089001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,011 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/3bf17ebc97244987b12878fe3a4c47e9 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/3bf17ebc97244987b12878fe3a4c47e9 2023-04-18 18:53:49,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:49,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:49,021 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:49,021 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:49,022 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:49,022 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:49,023 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,023 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,029 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:49,029 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:49,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:49,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:49,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844089023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,030 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,030 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:49,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,038 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/B of d1a51bf079ac6b54cf5bff45ab0a5c8e into 3bf17ebc97244987b12878fe3a4c47e9(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:49,038 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:49,039 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/B, priority=13, startTime=1681844028739; duration=0sec 2023-04-18 18:53:49,039 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:53:49,039 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:53:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:49,042 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 31510 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:53:49,042 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/C is initiating minor compaction (all files) 2023-04-18 18:53:49,042 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/C in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:49,042 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/fbd61286b90946398d594e64bb17d43f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/910a494e285645e9aa67a8a70e2c37d2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/1f6aac2d31204612899b019afcfd2b5c] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=30.8 K 2023-04-18 18:53:49,043 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting fbd61286b90946398d594e64bb17d43f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1681844025819 2023-04-18 18:53:49,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:49,043 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/9b54115a30e446458be790e24bfe7d17 2023-04-18 18:53:49,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:49,044 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:49,044 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:49,044 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 910a494e285645e9aa67a8a70e2c37d2, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1681844026492 2023-04-18 18:53:49,045 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 1f6aac2d31204612899b019afcfd2b5c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681844027560 2023-04-18 18:53:49,048 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:49,052 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:49,052 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:49,052 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:49,052 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:49,052 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:49,052 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:49,052 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:49,053 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:49,053 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,053 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:49,053 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:49,057 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,057 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:49,057 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,057 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:49,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:49,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:49,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:49,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:49,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10138c6a[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:49,061 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:49,061 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,066 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#C#compaction#26 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:49,069 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,070 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:49,070 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:49,070 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:49,070 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,071 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:49,073 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,073 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:49,073 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:49,073 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:49,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:49,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:49,094 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:49,094 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:49,094 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:49,094 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:49,094 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:49,103 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,103 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,104 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:49,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:49,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:49,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:49,106 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:49,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,109 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,113 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:49,113 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:49,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:49,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:49,113 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,113 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,114 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,114 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,114 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:49,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:49,114 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:49,114 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:49,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:49,113 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:49,114 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,115 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,118 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/61122ce2190c44a998b9f6c720f7358d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/61122ce2190c44a998b9f6c720f7358d 2023-04-18 18:53:49,128 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/C of d1a51bf079ac6b54cf5bff45ab0a5c8e into 61122ce2190c44a998b9f6c720f7358d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:49,128 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:49,128 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/C, priority=13, startTime=1681844028739; duration=0sec 2023-04-18 18:53:49,128 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:49,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:49,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844089203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,210 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:49,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844089209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:49,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844089212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,214 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:49,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:49,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844089212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:49,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:49,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:49,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:49,226 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:49,226 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:49,228 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:49,228 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:49,229 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,229 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,230 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:49,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:49,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:49,230 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,231 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,231 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:49,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:49,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844089233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:49,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:49,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:49,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:49,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:49,238 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:49,241 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:49,241 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:49,244 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:49,244 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:49,245 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:49,245 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:49,245 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,245 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:49,245 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:49,248 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,248 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:49,248 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,248 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:49,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:49,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:49,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:49,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:49,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c6d37a6[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:49,251 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:49,251 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,253 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,253 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:49,253 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,253 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:49,253 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:49,253 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:49,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:49,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:49,268 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:49,268 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:49,268 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:49,268 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:49,268 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:49,270 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,270 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:49,270 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:49,271 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:49,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:49,283 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,289 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:49,289 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:49,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:49,291 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,291 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,291 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,291 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,291 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,291 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,291 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:49,291 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:49,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:49,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:49,297 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:49,297 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:49,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:49,397 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:49,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:49,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:49,404 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/322a856f015b4679adb4dec2656ba67c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/322a856f015b4679adb4dec2656ba67c 2023-04-18 18:53:49,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:49,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:49,404 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:49,404 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:49,405 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:49,405 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:49,406 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,406 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,408 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,408 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:49,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:49,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:49,409 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,409 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:49,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:49,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:49,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:49,411 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:49,411 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:49,412 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:49,412 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:49,413 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:49,413 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:49,413 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:49,413 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:49,413 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:49,413 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:49,413 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:49,413 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,413 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:49,413 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:49,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,415 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:49,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:49,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:49,416 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/A of d1a51bf079ac6b54cf5bff45ab0a5c8e into 322a856f015b4679adb4dec2656ba67c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:49,416 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:49,416 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/A, priority=13, startTime=1681844028738; duration=0sec 2023-04-18 18:53:49,417 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:49,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:49,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:49,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b1e9fce[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:49,420 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:49,420 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,422 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,423 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:49,423 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:49,423 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:49,423 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,423 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:49,424 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,424 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:49,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:49,424 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:49,425 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:49,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:49,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:49,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:49,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:49,428 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:49,428 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:49,428 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:49,428 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:49,428 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:49,430 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,430 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:49,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:49,430 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:49,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,431 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:49,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,434 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:49,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,437 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,437 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,437 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,437 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,437 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,437 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:49,437 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,437 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:49,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:49,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:49,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:49,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:49,437 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,441 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:49,441 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:49,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:49,489 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/9b24d62a4494401fa1ded32be9a3a43f 2023-04-18 18:53:49,498 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/d17096b41b774bec8310286fceb4bd9a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/d17096b41b774bec8310286fceb4bd9a 2023-04-18 18:53:49,507 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/d17096b41b774bec8310286fceb4bd9a, entries=200, sequenceid=305, filesize=14.3 K 2023-04-18 18:53:49,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/9b54115a30e446458be790e24bfe7d17 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/9b54115a30e446458be790e24bfe7d17 2023-04-18 18:53:49,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:49,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844089510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:49,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844089513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:49,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844089516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:49,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844089516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,520 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/9b54115a30e446458be790e24bfe7d17, entries=150, sequenceid=305, filesize=11.9 K 2023-04-18 18:53:49,522 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/9b24d62a4494401fa1ded32be9a3a43f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/9b24d62a4494401fa1ded32be9a3a43f 2023-04-18 18:53:49,534 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/9b24d62a4494401fa1ded32be9a3a43f, entries=150, sequenceid=305, filesize=11.9 K 2023-04-18 18:53:49,536 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=93.93 KB/96180 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 747ms, sequenceid=305, compaction requested=false 2023-04-18 18:53:49,536 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:49,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:49,540 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-18 18:53:49,540 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:49,540 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:49,540 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:49,540 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:49,540 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:49,540 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:49,542 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:49,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:49,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:49,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:49,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:49,550 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:49,550 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:49,557 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:49,557 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:49,557 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,557 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,563 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/1f7c1572cdb14318b51c5ca39a2aa577 2023-04-18 18:53:49,564 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:49,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:49,567 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:49,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:49,570 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:49,570 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:49,572 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:49,572 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,572 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:49,572 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:49,572 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:49,572 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:49,572 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:49,572 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:49,573 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:49,573 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:49,573 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:49,573 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,573 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:49,573 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:49,574 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,574 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:49,575 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,575 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:49,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:49,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39d0280e[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:49,590 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:49,590 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,594 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,594 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:49,594 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:49,594 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:49,594 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,594 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:49,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:49,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:49,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:49,600 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:49,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:49,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:49,610 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:49,610 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:49,610 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:49,611 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:49,611 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:49,618 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,618 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:49,618 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:49,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:49,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:49,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:49,619 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,621 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:49,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:49,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,626 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:49,626 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:49,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:49,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:49,626 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,626 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,626 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,626 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:49,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:49,626 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:49,626 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,627 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:49,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:49,630 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:49,636 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:49,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:49,636 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844089626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,641 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/d8a9513073724a0abd9281335f861f53 2023-04-18 18:53:49,693 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/ab81b78a68e24cc69d69bbd30efab104 2023-04-18 18:53:49,704 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/1f7c1572cdb14318b51c5ca39a2aa577 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/1f7c1572cdb14318b51c5ca39a2aa577 2023-04-18 18:53:49,713 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/1f7c1572cdb14318b51c5ca39a2aa577, entries=150, sequenceid=326, filesize=11.9 K 2023-04-18 18:53:49,715 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/d8a9513073724a0abd9281335f861f53 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/d8a9513073724a0abd9281335f861f53 2023-04-18 18:53:49,723 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/d8a9513073724a0abd9281335f861f53, entries=150, sequenceid=326, filesize=11.9 K 2023-04-18 18:53:49,730 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/ab81b78a68e24cc69d69bbd30efab104 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/ab81b78a68e24cc69d69bbd30efab104 2023-04-18 18:53:49,736 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:49,738 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/ab81b78a68e24cc69d69bbd30efab104, entries=150, sequenceid=326, filesize=11.9 K 2023-04-18 18:53:49,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:49,745 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 205ms, sequenceid=326, compaction requested=true 2023-04-18 18:53:49,746 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:49,746 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:49,746 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:53:49,746 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:53:49,746 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-18 18:53:49,746 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:53:49,749 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36798 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:53:49,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:49,749 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/B is initiating minor compaction (all files) 2023-04-18 18:53:49,749 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/B in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:49,750 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/3bf17ebc97244987b12878fe3a4c47e9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/9b54115a30e446458be790e24bfe7d17, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/d8a9513073724a0abd9281335f861f53] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=35.9 K 2023-04-18 18:53:49,750 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:53:49,750 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/A is initiating minor compaction (all files) 2023-04-18 18:53:49,750 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/A in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:49,751 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/322a856f015b4679adb4dec2656ba67c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/d17096b41b774bec8310286fceb4bd9a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/1f7c1572cdb14318b51c5ca39a2aa577] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=38.3 K 2023-04-18 18:53:49,751 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 322a856f015b4679adb4dec2656ba67c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681844027560 2023-04-18 18:53:49,751 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 3bf17ebc97244987b12878fe3a4c47e9, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681844027560 2023-04-18 18:53:49,753 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-18 18:53:49,753 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:49,753 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:49,753 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:49,753 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:49,753 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:49,753 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:49,754 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting d17096b41b774bec8310286fceb4bd9a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1681844027639 2023-04-18 18:53:49,754 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 9b54115a30e446458be790e24bfe7d17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1681844027639 2023-04-18 18:53:49,755 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 1f7c1572cdb14318b51c5ca39a2aa577, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1681844028849 2023-04-18 18:53:49,755 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting d8a9513073724a0abd9281335f861f53, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1681844028849 2023-04-18 18:53:49,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:49,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:49,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:49,758 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:49,758 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:49,765 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:49,765 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:49,766 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,766 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,768 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,769 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:49,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:49,769 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,770 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:49,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:49,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:49,789 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:49,789 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:49,797 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:49,798 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#A#compaction#31 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:49,799 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#B#compaction#32 average throughput is 0.73 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:49,805 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:49,805 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:49,806 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:49,806 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:49,806 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:49,806 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:49,807 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:49,807 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:49,807 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,807 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:49,807 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:49,814 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,814 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:49,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:49,815 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,815 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:49,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:49,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35ac9382[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:49,818 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:49,818 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,820 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,820 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:49,820 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:49,820 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:49,820 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,820 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:49,822 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:49,822 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:49,822 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:49,822 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:49,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:49,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:49,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:49,826 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:49,826 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:49,826 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:49,826 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:49,826 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:49,828 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,828 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:49,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:49,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,828 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:49,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:49,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:49,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:49,830 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:49,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,831 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:49,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,836 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:49,836 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:49,836 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:49,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:49,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:49,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:49,836 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:49,836 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:49,837 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:49,836 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,837 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:49,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:49,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:49,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:49,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844089846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,871 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/d870df2417ac47ff87673ca3eddb892d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/d870df2417ac47ff87673ca3eddb892d 2023-04-18 18:53:49,881 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/A of d1a51bf079ac6b54cf5bff45ab0a5c8e into d870df2417ac47ff87673ca3eddb892d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:49,881 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:49,881 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/A, priority=13, startTime=1681844029746; duration=0sec 2023-04-18 18:53:49,882 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:53:49,882 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:53:49,885 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36798 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:53:49,885 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/C is initiating minor compaction (all files) 2023-04-18 18:53:49,885 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/C in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:49,885 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/61122ce2190c44a998b9f6c720f7358d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/9b24d62a4494401fa1ded32be9a3a43f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/ab81b78a68e24cc69d69bbd30efab104] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=35.9 K 2023-04-18 18:53:49,886 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 61122ce2190c44a998b9f6c720f7358d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681844027560 2023-04-18 18:53:49,887 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 9b24d62a4494401fa1ded32be9a3a43f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1681844027639 2023-04-18 18:53:49,887 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting ab81b78a68e24cc69d69bbd30efab104, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1681844028849 2023-04-18 18:53:49,899 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#C#compaction#33 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:49,937 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:49,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:49,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:49,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:49,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:49,947 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:49,947 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:49,949 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:49,949 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:49,949 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,949 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,951 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:49,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:49,951 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:49,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,952 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,953 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:49,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:49,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844089951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:49,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:49,957 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:49,957 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:49,961 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:49,961 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:49,961 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:49,962 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:49,962 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:49,962 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:49,962 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:49,962 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:49,962 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:49,962 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,962 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:49,962 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:49,965 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:49,966 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:49,967 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,967 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:49,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:49,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:49,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50195296[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:49,971 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:49,971 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,974 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,974 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:49,974 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:49,974 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:49,974 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,974 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:49,976 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:49,976 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:49,976 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:49,976 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:49,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:49,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:49,982 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:49,982 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:49,982 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:49,982 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:49,982 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:49,987 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,987 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:49,987 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:49,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:49,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,989 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,993 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:49,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,997 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,997 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:49,997 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:49,997 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,997 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:49,997 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:49,997 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,997 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:49,997 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:49,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:49,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:49,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:49,998 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:49,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:49,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:49,999 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:50,000 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:50,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:50,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:50,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844090019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844090020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:50,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844090022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:50,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844090023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,100 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:50,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:50,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:50,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:50,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:50,111 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:50,111 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:50,112 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:50,112 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:50,113 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,113 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,115 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:50,115 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:50,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:50,116 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,116 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:50,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,117 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:50,117 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:50,117 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:50,117 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:50,117 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:50,118 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:50,119 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:50,119 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:50,119 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:50,119 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:50,119 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:50,119 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:50,119 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:50,119 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:50,119 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,119 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:50,119 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:50,121 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,121 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:50,121 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,121 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:50,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:50,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fd647b4[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:50,123 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:50,124 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,125 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,125 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:50,125 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:50,125 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:50,125 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,125 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:50,126 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:50,126 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,126 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:50,127 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:50,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:50,132 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:50,132 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:50,132 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:50,132 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:50,132 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:50,138 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,138 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:50,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,138 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:50,138 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,140 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,146 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:50,146 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:50,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:50,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:50,146 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:50,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:50,146 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:50,147 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:50,147 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:50,146 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,147 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,147 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,147 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:50,147 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,147 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:50,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33410 deadline: 1681844090158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,204 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/761196f8f8a8427e8cc6e679f18acef0 2023-04-18 18:53:50,225 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/c149a5721f2e4127938aee1fe405fd24 2023-04-18 18:53:50,242 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/2bb995d1cb164024ad87cde65e8d5430 2023-04-18 18:53:50,247 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:50,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:50,249 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/761196f8f8a8427e8cc6e679f18acef0 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/761196f8f8a8427e8cc6e679f18acef0 2023-04-18 18:53:50,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:50,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:50,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:50,254 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:50,254 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:50,257 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:50,257 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:50,257 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,257 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,259 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,260 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:50,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,262 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,262 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:50,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:50,264 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:50,264 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:50,264 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:50,264 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:50,266 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:50,267 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:50,267 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:50,268 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:50,268 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:50,268 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:50,268 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:50,268 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:50,268 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:50,268 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,268 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:50,268 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:50,273 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/761196f8f8a8427e8cc6e679f18acef0, entries=200, sequenceid=341, filesize=14.3 K 2023-04-18 18:53:50,275 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,275 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:50,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:50,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/c149a5721f2e4127938aee1fe405fd24 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/c149a5721f2e4127938aee1fe405fd24 2023-04-18 18:53:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:50,283 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/a5fe6a1ea6a940fda0e76d95f8604b9b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/a5fe6a1ea6a940fda0e76d95f8604b9b 2023-04-18 18:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:50,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@477e9018[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:50,288 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:50,288 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,290 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,295 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/c149a5721f2e4127938aee1fe405fd24, entries=150, sequenceid=341, filesize=11.9 K 2023-04-18 18:53:50,293 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,296 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:50,296 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:50,296 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:50,296 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:50,297 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/2bb995d1cb164024ad87cde65e8d5430 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/2bb995d1cb164024ad87cde65e8d5430 2023-04-18 18:53:50,298 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:50,299 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,299 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:50,299 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,304 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/B of d1a51bf079ac6b54cf5bff45ab0a5c8e into a5fe6a1ea6a940fda0e76d95f8604b9b(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:50,304 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:50,304 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/B, priority=13, startTime=1681844029746; duration=0sec 2023-04-18 18:53:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:50,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:50,304 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:50,304 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:50,304 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:50,304 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:50,304 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:50,304 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:50,309 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,309 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:50,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,309 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/2bb995d1cb164024ad87cde65e8d5430, entries=150, sequenceid=341, filesize=11.9 K 2023-04-18 18:53:50,309 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:50,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:50,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:50,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:50,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:50,311 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 557ms, sequenceid=341, compaction requested=false 2023-04-18 18:53:50,311 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,311 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:50,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,312 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:50,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,315 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:50,315 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:50,315 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:50,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:50,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:50,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:50,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:50,315 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,317 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,317 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,317 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,317 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,317 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,317 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:50,317 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:50,315 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:50,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:50,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:50,331 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/7e35225f1c5d460d9e8b0662c2f2aada as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/7e35225f1c5d460d9e8b0662c2f2aada 2023-04-18 18:53:50,342 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/C of d1a51bf079ac6b54cf5bff45ab0a5c8e into 7e35225f1c5d460d9e8b0662c2f2aada(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:50,342 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:50,342 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/C, priority=13, startTime=1681844029746; duration=0sec 2023-04-18 18:53:50,342 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:50,417 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:50,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:50,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:50,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:50,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:50,424 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:50,424 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:50,428 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:50,428 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:50,430 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,430 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,433 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:50,433 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:50,433 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,433 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:50,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:50,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:50,436 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:50,436 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:50,439 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:50,439 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:50,439 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:50,439 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:50,439 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:50,439 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-18 18:53:50,440 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:50,440 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:50,440 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:50,440 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:50,440 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:50,440 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:50,464 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:50,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:50,497 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/fd52d78731a14e1ab81a8111b8c6930f 2023-04-18 18:53:50,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:50,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844090518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,522 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/65dfaf169bf54317918f577dffb63cbb 2023-04-18 18:53:50,543 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/46a9fbc8185946f3a3fc51a382ecbc3a 2023-04-18 18:53:50,550 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/fd52d78731a14e1ab81a8111b8c6930f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/fd52d78731a14e1ab81a8111b8c6930f 2023-04-18 18:53:50,558 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/fd52d78731a14e1ab81a8111b8c6930f, entries=150, sequenceid=361, filesize=11.9 K 2023-04-18 18:53:50,560 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/65dfaf169bf54317918f577dffb63cbb as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/65dfaf169bf54317918f577dffb63cbb 2023-04-18 18:53:50,568 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/65dfaf169bf54317918f577dffb63cbb, entries=150, sequenceid=361, filesize=11.9 K 2023-04-18 18:53:50,570 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/46a9fbc8185946f3a3fc51a382ecbc3a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/46a9fbc8185946f3a3fc51a382ecbc3a 2023-04-18 18:53:50,577 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/46a9fbc8185946f3a3fc51a382ecbc3a, entries=150, sequenceid=361, filesize=11.9 K 2023-04-18 18:53:50,579 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 140ms, sequenceid=361, compaction requested=true 2023-04-18 18:53:50,579 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:50,579 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:50,579 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:50,586 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:50,586 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,586 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:50,586 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:50,591 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,591 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:50,592 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,592 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:50,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:50,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:50,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:50,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@169f3bf6[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:50,595 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:50,595 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,597 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,597 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:50,597 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,598 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:50,597 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:50,598 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:50,600 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:50,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:50,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:50,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:50,604 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:50,604 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:50,604 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:50,604 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:50,605 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:50,606 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:50,606 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,607 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:50,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:50,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,607 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,609 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,614 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:50,614 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:50,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:50,614 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,614 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,614 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,614 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:50,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:50,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:50,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:50,614 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,618 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,618 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,618 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,618 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:50,618 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:50,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:50,624 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-18 18:53:50,624 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:50,625 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:50,625 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:50,625 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:50,625 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:50,625 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:50,657 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/6c39866de56647c0b16eefd0af916b97 2023-04-18 18:53:50,690 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/04ba904878e84ccbbe908f53b1172ef3 2023-04-18 18:53:50,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:50,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844090692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,718 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:50,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:50,721 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/35cff34ae4594e9c884864b85b999bd6 2023-04-18 18:53:50,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:50,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:50,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:50,726 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:50,726 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:50,726 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:50,726 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:50,727 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,727 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,729 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:50,729 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:50,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,730 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,730 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,731 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:50,731 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/6c39866de56647c0b16eefd0af916b97 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/6c39866de56647c0b16eefd0af916b97 2023-04-18 18:53:50,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:50,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:50,732 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:50,732 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:50,732 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:50,744 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/6c39866de56647c0b16eefd0af916b97, entries=200, sequenceid=377, filesize=14.3 K 2023-04-18 18:53:50,747 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:50,747 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:50,747 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:50,747 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:50,747 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/04ba904878e84ccbbe908f53b1172ef3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/04ba904878e84ccbbe908f53b1172ef3 2023-04-18 18:53:50,747 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:50,747 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:50,748 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:50,748 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:50,748 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,748 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:50,748 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:50,755 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,755 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:50,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:50,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:50,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:50,765 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/04ba904878e84ccbbe908f53b1172ef3, entries=150, sequenceid=377, filesize=11.9 K 2023-04-18 18:53:50,765 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:50,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c085651[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:50,765 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,766 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/35cff34ae4594e9c884864b85b999bd6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/35cff34ae4594e9c884864b85b999bd6 2023-04-18 18:53:50,767 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,767 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,767 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:50,767 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:50,767 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:50,767 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:50,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:50,777 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:50,777 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:50,777 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:50,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:50,783 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:50,783 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:50,783 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:50,783 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:50,783 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:50,784 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/35cff34ae4594e9c884864b85b999bd6, entries=150, sequenceid=377, filesize=11.9 K 2023-04-18 18:53:50,785 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 161ms, sequenceid=377, compaction requested=true 2023-04-18 18:53:50,786 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:50,786 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:53:50,786 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:53:50,786 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:53:50,786 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:50,786 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,786 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:50,787 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:50,787 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:53:50,787 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,788 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49086 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:53:50,788 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/B is initiating minor compaction (all files) 2023-04-18 18:53:50,788 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/B in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:50,789 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/a5fe6a1ea6a940fda0e76d95f8604b9b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/c149a5721f2e4127938aee1fe405fd24, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/65dfaf169bf54317918f577dffb63cbb, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/04ba904878e84ccbbe908f53b1172ef3] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=47.9 K 2023-04-18 18:53:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,789 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,790 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting a5fe6a1ea6a940fda0e76d95f8604b9b, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1681844028849 2023-04-18 18:53:50,790 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53966 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:53:50,793 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/A is initiating minor compaction (all files) 2023-04-18 18:53:50,794 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/A in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:50,794 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/d870df2417ac47ff87673ca3eddb892d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/761196f8f8a8427e8cc6e679f18acef0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/fd52d78731a14e1ab81a8111b8c6930f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/6c39866de56647c0b16eefd0af916b97] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=52.7 K 2023-04-18 18:53:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:50,794 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting c149a5721f2e4127938aee1fe405fd24, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1681844029574 2023-04-18 18:53:50,796 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting d870df2417ac47ff87673ca3eddb892d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1681844028849 2023-04-18 18:53:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,796 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 65dfaf169bf54317918f577dffb63cbb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1681844029833 2023-04-18 18:53:50,797 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 761196f8f8a8427e8cc6e679f18acef0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1681844029574 2023-04-18 18:53:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,797 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 04ba904878e84ccbbe908f53b1172ef3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681844030484 2023-04-18 18:53:50,798 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting fd52d78731a14e1ab81a8111b8c6930f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1681844029833 2023-04-18 18:53:50,799 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 6c39866de56647c0b16eefd0af916b97, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681844030484 2023-04-18 18:53:50,804 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:50,804 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:50,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:50,805 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,805 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,805 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,805 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,805 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,805 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:50,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:50,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:50,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:50,805 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-18 18:53:50,806 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:50,806 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:50,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:50,805 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,805 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:50,806 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:50,806 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:50,806 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:50,806 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:50,806 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:50,806 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:50,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:50,835 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#A#compaction#43 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:50,840 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#B#compaction#44 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:50,864 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/0486197ad40444f8a9150a5175bba25a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/0486197ad40444f8a9150a5175bba25a 2023-04-18 18:53:50,899 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/A of d1a51bf079ac6b54cf5bff45ab0a5c8e into 0486197ad40444f8a9150a5175bba25a(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:50,899 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:50,899 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/A, priority=12, startTime=1681844030786; duration=0sec 2023-04-18 18:53:50,899 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:53:50,899 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:53:50,904 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49086 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:53:50,904 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/C is initiating minor compaction (all files) 2023-04-18 18:53:50,904 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/C in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:50,904 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/7e35225f1c5d460d9e8b0662c2f2aada, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/2bb995d1cb164024ad87cde65e8d5430, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/46a9fbc8185946f3a3fc51a382ecbc3a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/35cff34ae4594e9c884864b85b999bd6] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=47.9 K 2023-04-18 18:53:50,905 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 7e35225f1c5d460d9e8b0662c2f2aada, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1681844028849 2023-04-18 18:53:50,906 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 2bb995d1cb164024ad87cde65e8d5430, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1681844029574 2023-04-18 18:53:50,906 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:50,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:50,908 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 46a9fbc8185946f3a3fc51a382ecbc3a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1681844029833 2023-04-18 18:53:50,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:50,912 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 35cff34ae4594e9c884864b85b999bd6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681844030484 2023-04-18 18:53:50,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:50,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:50,913 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:50,913 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:50,913 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:50,913 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:50,914 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,914 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:50,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844090917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,921 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:50,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:50,921 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:50,922 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,922 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:50,924 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:50,924 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:50,924 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:50,924 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:50,924 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:50,924 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:50,925 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:50,925 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:50,925 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:50,925 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:50,925 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:50,925 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,925 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:50,925 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:50,927 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,927 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:50,928 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,928 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,929 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#C#compaction#45 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:50,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:50,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41ae5b83[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:50,931 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:50,931 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,933 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,933 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:50,933 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:50,933 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:50,933 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,933 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:50,935 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,935 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:50,935 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:50,935 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:50,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:50,939 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:50,939 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:50,939 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:50,939 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:50,939 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:50,941 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,941 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:50,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,941 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:50,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:50,942 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,947 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,950 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,950 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:50,950 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:50,950 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:50,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:50,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:50,950 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:50,951 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:50,951 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,951 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:50,951 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:50,950 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:50,951 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:50,951 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:50,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:50,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:50,955 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/f1ee4e644cb84232a44db82a6ab71450 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/f1ee4e644cb84232a44db82a6ab71450 2023-04-18 18:53:50,964 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/C of d1a51bf079ac6b54cf5bff45ab0a5c8e into f1ee4e644cb84232a44db82a6ab71450(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:50,964 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:50,964 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/C, priority=12, startTime=1681844030787; duration=0sec 2023-04-18 18:53:50,964 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:51,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:51,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844091022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:51,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844091026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:51,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844091029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:51,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844091033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:51,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844091036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,051 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:51,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:51,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:51,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:51,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:51,057 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:51,057 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:51,057 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:51,057 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:51,057 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,058 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,061 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:51,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:51,062 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,062 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,063 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:51,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:51,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:51,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:51,064 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:51,065 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:51,065 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:51,065 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:51,065 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:51,065 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:51,065 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,065 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:51,065 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:51,068 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:51,069 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,069 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:51,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:51,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7263fdfe[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:51,074 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:51,074 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,075 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,075 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:51,076 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:51,076 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:51,077 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,077 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:51,079 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:51,079 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:51,080 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:51,080 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:51,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:51,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:51,090 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:51,090 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:51,090 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:51,090 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:51,090 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:51,093 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,093 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,093 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:51,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,096 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:51,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,104 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:51,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,107 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:51,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:51,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:51,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:51,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:51,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:51,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:51,109 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:51,109 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:51,109 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:51,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:51,209 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:51,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:51,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:51,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:51,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:51,216 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:51,216 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:51,216 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:51,216 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:51,217 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,217 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,218 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,219 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:51,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:51,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:51,219 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,219 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:51,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:51,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:51,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:51,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:51,224 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:51,224 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:51,224 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:51,226 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:51,226 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:51,226 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:51,226 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:51,227 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:51,227 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:51,227 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:51,227 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:51,227 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,227 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:51,227 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:51,229 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,229 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:51,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:51,229 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,229 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:51,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844091225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:51,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:51,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dcc38cf[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:51,234 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:51,234 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,237 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:51,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:51,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:51,237 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,238 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:51,239 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:51,239 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,239 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:51,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:51,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:51,239 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:51,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:51,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:51,244 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:51,244 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:51,244 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:51,244 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:51,244 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:51,246 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,246 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:51,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,246 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:51,247 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:51,252 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:51,261 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:51,261 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:51,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:51,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:51,264 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,264 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,264 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:51,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:51,264 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:51,264 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/33c34b9c131b4e8bbb491b6d2b8df3c8 2023-04-18 18:53:51,264 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,264 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:51,265 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:51,265 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,265 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,265 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:51,325 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/4f8103a65da84e05995d0ee6f89ba63a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/4f8103a65da84e05995d0ee6f89ba63a 2023-04-18 18:53:51,336 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/B of d1a51bf079ac6b54cf5bff45ab0a5c8e into 4f8103a65da84e05995d0ee6f89ba63a(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:51,336 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:51,336 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/B, priority=12, startTime=1681844030786; duration=0sec 2023-04-18 18:53:51,337 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:51,365 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:51,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:51,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:51,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:51,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:51,370 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:51,370 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:51,370 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:51,370 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:51,370 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,371 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,372 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:51,372 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:51,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,373 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,373 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:51,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:51,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:51,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:51,374 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:51,374 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:51,374 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:51,376 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:51,376 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:51,379 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:51,379 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:51,379 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:51,379 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:51,379 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:51,379 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:51,379 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,379 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:51,379 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:51,382 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,382 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:51,385 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,385 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:51,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:51,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53384ff9[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:51,387 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:51,387 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,389 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,389 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:51,389 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:51,389 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:51,391 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,391 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:51,392 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:51,392 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:51,392 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:51,392 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:51,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:51,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:51,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:51,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:51,396 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:51,396 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:51,396 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:51,396 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:51,396 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:51,398 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,398 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:51,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,398 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:51,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:51,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,399 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:51,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,403 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:51,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,409 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:51,409 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:51,409 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,409 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,409 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:51,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:51,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:51,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:51,409 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,409 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,410 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,410 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,410 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:51,410 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:51,410 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:51,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:51,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:51,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:51,510 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:51,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:51,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:51,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:51,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:51,519 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:51,519 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:51,520 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:51,520 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:51,520 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,520 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,522 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:51,523 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:51,523 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,523 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:51,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:51,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:51,525 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:51,525 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:51,525 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:51,525 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:51,525 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:51,525 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:51,525 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:51,525 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:51,525 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:51,526 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:51,526 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:51,526 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,526 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:51,526 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:51,527 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,527 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:51,528 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,528 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:51,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:51,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6079645e[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:51,530 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:51,530 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,533 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,533 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:51,533 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:51,533 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:51,533 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,533 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:51,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:51,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844091532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,535 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:51,535 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:51,535 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:51,536 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:51,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:51,542 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:51,542 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:51,542 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:51,542 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:51,542 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:51,546 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,546 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:51,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,546 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:51,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:51,548 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,551 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,554 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,554 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,554 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:51,554 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:51,555 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,555 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,555 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,555 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,555 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:51,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:51,555 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:51,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:51,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:51,556 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:51,556 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:51,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:51,656 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:51,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:51,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:51,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:51,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:51,661 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:51,661 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:51,661 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:51,661 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:51,662 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,662 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,664 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,664 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:51,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:51,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:51,664 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,664 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:51,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:51,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:51,666 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:51,666 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:51,666 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:51,676 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:51,676 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:51,676 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:51,677 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:51,677 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:51,677 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:51,677 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,677 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:51,677 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:51,679 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,679 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,679 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:51,679 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:51,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:51,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d98f9a2[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:51,688 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:51,688 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,690 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,690 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,690 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:51,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,690 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:51,690 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:51,690 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:51,692 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:51,692 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:51,692 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:51,693 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:51,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:51,698 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:51,698 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:51,698 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:51,698 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:51,698 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:51,701 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,701 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:51,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:51,702 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,702 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,705 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:51,707 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:51,708 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:51,708 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:51,708 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:51,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:51,708 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:51,708 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,708 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,708 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,708 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:51,708 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,708 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:51,708 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:51,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:51,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:51,740 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/17a9763e5afd49ef8029a260ebce8c9c 2023-04-18 18:53:51,761 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/afb04eaea7bb453e9b1743e7534dff80 2023-04-18 18:53:51,770 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/33c34b9c131b4e8bbb491b6d2b8df3c8 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/33c34b9c131b4e8bbb491b6d2b8df3c8 2023-04-18 18:53:51,778 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/33c34b9c131b4e8bbb491b6d2b8df3c8, entries=200, sequenceid=394, filesize=14.3 K 2023-04-18 18:53:51,780 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/17a9763e5afd49ef8029a260ebce8c9c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/17a9763e5afd49ef8029a260ebce8c9c 2023-04-18 18:53:51,787 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/17a9763e5afd49ef8029a260ebce8c9c, entries=150, sequenceid=394, filesize=11.9 K 2023-04-18 18:53:51,788 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/afb04eaea7bb453e9b1743e7534dff80 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/afb04eaea7bb453e9b1743e7534dff80 2023-04-18 18:53:51,797 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/afb04eaea7bb453e9b1743e7534dff80, entries=150, sequenceid=394, filesize=11.9 K 2023-04-18 18:53:51,798 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 993ms, sequenceid=394, compaction requested=false 2023-04-18 18:53:51,798 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:51,808 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:51,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:51,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:51,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:51,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:51,814 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:51,814 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:51,814 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:51,814 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:51,814 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,814 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,816 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,816 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:51,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:51,816 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:51,816 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:51,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:51,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:51,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:51,818 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:51,818 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:51,818 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:51,818 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:51,818 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:51,818 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:51,818 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:51,819 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-18 18:53:51,819 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:51,819 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:51,819 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:51,819 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:51,820 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:51,820 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:51,853 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=412 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/33bc1a2bd9e049a999463b3b36eb61d3 2023-04-18 18:53:52,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:52,041 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:52,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:52,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844092102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:52,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844092209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,287 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=412 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/ec2b7ecee7824dea8be53596f821db23 2023-04-18 18:53:52,308 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=412 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/71d72e0f42854b5c81df660f1b7d9a92 2023-04-18 18:53:52,318 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/33bc1a2bd9e049a999463b3b36eb61d3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/33bc1a2bd9e049a999463b3b36eb61d3 2023-04-18 18:53:52,327 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/33bc1a2bd9e049a999463b3b36eb61d3, entries=150, sequenceid=412, filesize=11.9 K 2023-04-18 18:53:52,330 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/ec2b7ecee7824dea8be53596f821db23 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/ec2b7ecee7824dea8be53596f821db23 2023-04-18 18:53:52,338 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/ec2b7ecee7824dea8be53596f821db23, entries=150, sequenceid=412, filesize=11.9 K 2023-04-18 18:53:52,339 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/71d72e0f42854b5c81df660f1b7d9a92 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/71d72e0f42854b5c81df660f1b7d9a92 2023-04-18 18:53:52,352 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/71d72e0f42854b5c81df660f1b7d9a92, entries=150, sequenceid=412, filesize=11.9 K 2023-04-18 18:53:52,354 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 534ms, sequenceid=412, compaction requested=true 2023-04-18 18:53:52,354 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:52,354 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:52,354 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:52,362 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,362 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:52,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:52,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:52,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,364 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:52,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:52,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@716ea29e[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:52,365 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:52,365 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,367 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,367 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:52,367 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:52,367 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:52,367 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,367 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:52,370 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,370 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:52,370 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:52,370 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:52,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:52,376 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:52,376 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:52,376 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:52,376 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:52,376 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:52,383 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,383 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,383 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,385 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:52,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,387 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:52,389 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,389 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,389 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,389 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:52,389 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:52,389 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:52,389 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,389 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:52,389 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,390 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:52,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:52,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:52,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:52,390 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:52,390 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:52,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:52,414 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-18 18:53:52,414 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:52,414 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:52,414 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:52,414 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:52,414 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:52,414 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:52,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:52,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844092456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,490 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:52,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:52,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:52,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:52,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:52,495 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:52,495 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:52,495 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:52,495 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:52,495 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,495 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,498 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:52,498 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:52,498 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,498 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:52,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:52,499 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:52,500 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:52,500 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:52,500 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:52,500 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:52,502 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:52,503 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:52,503 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,503 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:52,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:52,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60386ccd[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:52,505 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:52,505 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,506 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,507 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:52,507 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:52,507 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:52,507 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,507 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:52,509 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,509 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:52,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:52,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:52,509 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:52,509 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:52,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:52,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:52,513 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:52,513 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:52,513 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:52,513 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:52,513 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:52,518 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,518 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:52,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,518 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:52,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:52,519 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,520 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,525 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,525 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:52,525 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,525 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:52,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:52,525 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:52,525 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:52,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:52,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:52,525 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,525 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,525 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,525 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:52,526 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:52,526 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:52,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:52,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844092559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,626 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:52,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:52,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:52,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:52,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:52,630 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:52,631 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:52,631 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:52,631 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:52,631 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,631 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,632 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,632 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:52,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:52,633 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,633 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:52,635 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:52,635 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:52,635 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:52,635 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:52,635 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:52,635 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:52,636 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:52,636 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:52,636 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:52,636 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:52,636 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:52,636 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:52,636 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:52,636 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,636 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:52,636 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:52,638 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,638 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:52,638 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,638 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:52,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:52,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b48af00[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:52,640 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:52,640 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,646 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,646 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,646 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:52,646 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:52,646 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:52,646 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:52,648 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:52,648 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,648 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:52,648 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:52,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:52,650 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:52,651 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:52,651 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:52,651 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:52,651 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:52,652 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,652 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,652 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,653 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:52,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,654 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,657 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:52,657 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:52,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:52,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:52,657 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,657 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,657 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,657 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,657 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,657 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,657 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:52,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:52,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:52,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:52,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:52,657 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:52,658 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:52,658 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:52,758 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:52,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:52,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:52,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:52,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:52,763 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:52,763 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:52,763 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:52,763 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:52,763 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,764 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,766 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,766 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:52,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:52,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:52,766 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,766 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:52,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:52,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:52,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:52,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:52,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:52,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844092762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,769 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:52,769 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:52,769 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:52,769 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:52,769 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:52,769 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:52,769 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:52,769 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:52,769 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:52,769 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:52,769 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:52,769 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,769 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:52,769 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:52,772 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,772 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:52,772 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:52,772 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:52,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:52,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1571a22[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:52,780 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:52,781 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,784 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,785 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,785 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:52,785 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:52,785 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:52,785 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:52,786 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:52,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:52,787 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:52,787 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:52,787 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:52,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:52,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:52,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:52,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:52,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:52,790 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:52,790 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:52,790 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:52,790 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:52,790 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:52,792 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,792 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:52,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,792 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:52,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:52,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:52,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:52,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:52,794 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:52,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,803 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,806 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,806 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,806 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:52,806 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,806 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:52,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:52,806 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,807 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,807 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,807 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,807 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:52,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:52,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:52,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:52,807 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:52,807 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:52,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:52,832 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/38bad8a3ed304cbab273421d6faa5c91 2023-04-18 18:53:52,907 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:52,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:52,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:52,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:52,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:52,912 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:52,912 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:52,912 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:52,912 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:52,912 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,912 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,914 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,914 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:52,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:52,914 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,914 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:52,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:52,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:52,916 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:52,916 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:52,916 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:52,916 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:52,916 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:52,916 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:52,916 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:52,916 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:52,916 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:52,916 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:52,916 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:52,916 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,916 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:52,916 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:52,918 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,918 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:52,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:52,920 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,920 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:52,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52932c4[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:52,920 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:52,920 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,922 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,922 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,922 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:52,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,922 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:52,922 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:52,922 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:52,925 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,925 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:52,925 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:52,925 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:52,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:52,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:52,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:52,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:52,932 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:52,932 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:52,932 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:52,932 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:52,932 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:52,934 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,934 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,934 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:52,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:52,936 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,937 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,939 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,939 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,939 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:52,939 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:52,939 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,939 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:52,939 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:52,940 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:52,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:52,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:52,940 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:52,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:52,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:52,940 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:52,940 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:52,940 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:52,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:53,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:53,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844093034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,038 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4192 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:53:53,040 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:53,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:53,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:53,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844093041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:53,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844093045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,053 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:53:53,054 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:53:53,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:53,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:53,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:53,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844093054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:53,069 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:53,069 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:53,069 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:53,069 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:53,071 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4208 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:53:53,072 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,072 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:53,073 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,073 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:53,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:53,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:53,077 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,077 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:53,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844093072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:53,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:53,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:53,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:53,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:53,080 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:53,080 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:53,080 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:53,080 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:53,080 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:53,080 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:53,080 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:53,081 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:53,081 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:53,081 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:53,081 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:53,081 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,081 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:53,081 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:53,085 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:53,085 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:53,085 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,085 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:53,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:53,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26d468a8[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:53,087 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:53,087 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,089 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:53,092 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,092 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:53,093 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:53,093 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:53,093 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:53,093 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:53,097 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:53,097 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:53,097 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:53,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:53,097 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:53,097 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:53,099 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,099 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:53,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,100 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,102 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:53,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:53,104 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:53,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:53,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:53,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:53,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:53,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:53,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:53,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:53,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:53,108 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:53,109 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:53,109 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:53,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:53,209 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:53,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:53,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:53,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:53,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:53,225 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:53,225 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:53,225 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:53,225 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:53,226 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,226 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:53,227 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:53,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:53,228 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:53,228 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,229 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:53,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:53,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:53,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:53,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:53,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:53,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:53,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:53,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:53,233 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:53,233 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:53,240 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:53,240 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:53,240 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:53,240 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:53,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:53,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:53,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:53,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:53,242 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,242 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:53,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:53,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:53,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:53,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:53,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:53,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:53,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28eae3dd[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:53,247 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:53,247 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,248 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,248 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:53,249 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,249 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,249 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:53,252 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/4a1d8f207b8746d49e9f4fd0aedf5e60 2023-04-18 18:53:53,255 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:53,255 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:53,255 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:53,261 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,261 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:53,261 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:53,261 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:53,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:53,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:53,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:53,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:53,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:53,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:53,265 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:53,265 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:53,266 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:53,266 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:53,266 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:53,268 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,268 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:53,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,269 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:53,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:53,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:53,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,270 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:53,271 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:53,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,279 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:53,279 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:53,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:53,279 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,279 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:53,279 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:53,279 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,280 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:53,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:53,280 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,280 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:53,280 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:53,280 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:53,280 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:53,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:53,290 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/1cf4f303a2fd4a78ab93f0a98cbea1e3 2023-04-18 18:53:53,300 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/38bad8a3ed304cbab273421d6faa5c91 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/38bad8a3ed304cbab273421d6faa5c91 2023-04-18 18:53:53,307 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/38bad8a3ed304cbab273421d6faa5c91, entries=200, sequenceid=430, filesize=14.3 K 2023-04-18 18:53:53,309 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/4a1d8f207b8746d49e9f4fd0aedf5e60 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/4a1d8f207b8746d49e9f4fd0aedf5e60 2023-04-18 18:53:53,331 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/4a1d8f207b8746d49e9f4fd0aedf5e60, entries=150, sequenceid=430, filesize=11.9 K 2023-04-18 18:53:53,333 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/1cf4f303a2fd4a78ab93f0a98cbea1e3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/1cf4f303a2fd4a78ab93f0a98cbea1e3 2023-04-18 18:53:53,339 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/1cf4f303a2fd4a78ab93f0a98cbea1e3, entries=150, sequenceid=430, filesize=11.9 K 2023-04-18 18:53:53,341 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 927ms, sequenceid=430, compaction requested=true 2023-04-18 18:53:53,341 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:53,341 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:53,341 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:53:53,341 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:53:53,341 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:53,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:53:53,343 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54102 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:53:53,343 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/A is initiating minor compaction (all files) 2023-04-18 18:53:53,343 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/A in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:53,344 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/0486197ad40444f8a9150a5175bba25a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/33c34b9c131b4e8bbb491b6d2b8df3c8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/33bc1a2bd9e049a999463b3b36eb61d3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/38bad8a3ed304cbab273421d6faa5c91] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=52.8 K 2023-04-18 18:53:53,348 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 0486197ad40444f8a9150a5175bba25a, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681844030484 2023-04-18 18:53:53,348 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49222 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:53:53,348 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/B is initiating minor compaction (all files) 2023-04-18 18:53:53,349 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/B in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:53,349 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/4f8103a65da84e05995d0ee6f89ba63a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/17a9763e5afd49ef8029a260ebce8c9c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/ec2b7ecee7824dea8be53596f821db23, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/4a1d8f207b8746d49e9f4fd0aedf5e60] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=48.1 K 2023-04-18 18:53:53,349 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 33c34b9c131b4e8bbb491b6d2b8df3c8, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1681844030646 2023-04-18 18:53:53,349 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 4f8103a65da84e05995d0ee6f89ba63a, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681844030484 2023-04-18 18:53:53,350 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 33bc1a2bd9e049a999463b3b36eb61d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=412, earliestPutTs=1681844030866 2023-04-18 18:53:53,350 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 38bad8a3ed304cbab273421d6faa5c91, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1681844032068 2023-04-18 18:53:53,351 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 17a9763e5afd49ef8029a260ebce8c9c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1681844030683 2023-04-18 18:53:53,352 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting ec2b7ecee7824dea8be53596f821db23, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=412, earliestPutTs=1681844030866 2023-04-18 18:53:53,352 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 4a1d8f207b8746d49e9f4fd0aedf5e60, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1681844032096 2023-04-18 18:53:53,378 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#A#compaction#54 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:53,380 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:53,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:53,382 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#B#compaction#55 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:53,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:53,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:53,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:53,404 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:53,404 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:53,404 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:53,404 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:53,404 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,405 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:53,406 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,406 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:53,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:53,407 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,407 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:53,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:53,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:53,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:53,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:53,412 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:53,412 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:53,412 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:53,420 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:53,420 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:53,433 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:53,433 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:53,433 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-18 18:53:53,434 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:53,434 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:53,434 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:53,434 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:53,434 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:53,434 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:53,467 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/51622548e0db47bfb5b0b6ef40d8b57a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/51622548e0db47bfb5b0b6ef40d8b57a 2023-04-18 18:53:53,473 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/b3e68a88061746e89b66653d29fa38d2 2023-04-18 18:53:53,486 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/B of d1a51bf079ac6b54cf5bff45ab0a5c8e into 51622548e0db47bfb5b0b6ef40d8b57a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:53,486 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:53,486 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/B, priority=12, startTime=1681844033341; duration=0sec 2023-04-18 18:53:53,486 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:53:53,486 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:53:53,496 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49222 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:53:53,496 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/C is initiating minor compaction (all files) 2023-04-18 18:53:53,496 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/C in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:53,497 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/f1ee4e644cb84232a44db82a6ab71450, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/afb04eaea7bb453e9b1743e7534dff80, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/71d72e0f42854b5c81df660f1b7d9a92, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/1cf4f303a2fd4a78ab93f0a98cbea1e3] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=48.1 K 2023-04-18 18:53:53,498 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting f1ee4e644cb84232a44db82a6ab71450, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681844030484 2023-04-18 18:53:53,500 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting afb04eaea7bb453e9b1743e7534dff80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1681844030683 2023-04-18 18:53:53,500 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 71d72e0f42854b5c81df660f1b7d9a92, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=412, earliestPutTs=1681844030866 2023-04-18 18:53:53,501 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 1cf4f303a2fd4a78ab93f0a98cbea1e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1681844032096 2023-04-18 18:53:53,520 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#C#compaction#58 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:53,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:53,583 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:53,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:53,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844093657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:53,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844093764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,811 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/3554697fb9704cfbb6574fb9a239bae6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/3554697fb9704cfbb6574fb9a239bae6 2023-04-18 18:53:53,818 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/A of d1a51bf079ac6b54cf5bff45ab0a5c8e into 3554697fb9704cfbb6574fb9a239bae6(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:53,818 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:53,818 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/A, priority=12, startTime=1681844033341; duration=0sec 2023-04-18 18:53:53,818 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:53,923 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/fd002fc60cd7419a8b666005babd5356 2023-04-18 18:53:53,938 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/3eb57fcb666942d2a405772f88163607 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/3eb57fcb666942d2a405772f88163607 2023-04-18 18:53:53,942 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/5ee2eb17fa3046cba2f0a664e16e472e 2023-04-18 18:53:53,945 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/C of d1a51bf079ac6b54cf5bff45ab0a5c8e into 3eb57fcb666942d2a405772f88163607(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:53,945 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:53,945 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/C, priority=12, startTime=1681844033342; duration=0sec 2023-04-18 18:53:53,946 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:53,949 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/b3e68a88061746e89b66653d29fa38d2 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/b3e68a88061746e89b66653d29fa38d2 2023-04-18 18:53:53,954 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/b3e68a88061746e89b66653d29fa38d2, entries=150, sequenceid=444, filesize=11.9 K 2023-04-18 18:53:53,955 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/fd002fc60cd7419a8b666005babd5356 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/fd002fc60cd7419a8b666005babd5356 2023-04-18 18:53:53,962 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/fd002fc60cd7419a8b666005babd5356, entries=150, sequenceid=444, filesize=11.9 K 2023-04-18 18:53:53,964 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/5ee2eb17fa3046cba2f0a664e16e472e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/5ee2eb17fa3046cba2f0a664e16e472e 2023-04-18 18:53:53,969 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/5ee2eb17fa3046cba2f0a664e16e472e, entries=150, sequenceid=444, filesize=11.9 K 2023-04-18 18:53:53,970 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 537ms, sequenceid=444, compaction requested=false 2023-04-18 18:53:53,970 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:53,970 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:53,971 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:53,971 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:53,971 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,971 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:53,971 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:53,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:53,971 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-18 18:53:53,971 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:53,971 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:53,971 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:53,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:53,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:53,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:53,973 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:53,974 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:53,974 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,974 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:53,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:53,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:53,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:53,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11d57ae8[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:53,976 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:53,976 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,977 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,977 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,978 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:53,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:53,978 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:53,978 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:53,978 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:53,980 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:53,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:53,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:53,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:53,984 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:53,984 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:53,984 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:53,984 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:53,984 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:53,986 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,986 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:53,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,987 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:53,987 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/2f08a54c9c4944fe951b8c09d202c3d0 2023-04-18 18:53:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:53,989 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:53,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:53,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,991 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,007 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,007 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:54,007 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:54,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:54,007 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:54,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:54,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:54,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:54,007 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:54,007 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:54,007 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:54,007 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,007 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:54,008 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,008 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:54,008 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:54,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844094021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,108 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:54,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:54,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:54,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:54,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844094124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:54,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:54,132 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:54,132 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:54,132 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:54,132 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:54,133 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,133 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:54,134 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,134 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:54,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:54,135 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:54,135 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:54,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:54,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:54,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:54,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:54,136 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:54,136 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:54,137 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:54,137 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:54,137 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:54,137 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:54,137 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:54,137 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:54,137 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:54,137 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:54,137 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:54,137 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,137 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:54,137 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:54,139 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,139 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:54,139 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,139 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:54,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:54,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:54,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:54,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:54,141 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:54,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d26d504[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:54,141 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,142 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:54,143 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,143 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:54,145 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:54,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:54,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:54,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:54,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:54,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:54,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:54,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:54,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:54,149 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:54,149 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:54,150 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:54,150 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:54,150 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:54,152 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,152 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,152 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:54,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:54,153 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:54,155 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:54,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:54,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,161 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:54,161 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:54,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:54,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:54,166 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,166 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:54,166 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:54,166 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,166 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,166 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,166 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,167 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:54,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:54,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:54,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:54,167 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:54,167 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:54,267 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:54,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:54,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:54,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:54,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:54,272 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:54,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:54,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:54,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:54,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,273 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:54,274 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:54,274 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:54,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:54,275 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:54,275 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:54,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:54,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:54,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:54,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:54,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:54,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:54,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:54,277 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:54,277 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:54,277 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:54,277 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:54,277 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:54,277 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:54,277 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:54,277 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:54,277 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,277 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:54,277 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:54,283 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,283 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,283 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:54,288 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:54,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:54,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58b26720[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:54,292 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:54,292 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,293 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,293 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:54,294 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:54,294 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:54,294 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,294 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:54,296 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,296 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:54,296 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:54,296 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:54,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:54,299 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:54,299 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:54,299 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:54,299 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:54,299 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:54,301 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,301 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:54,301 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:54,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:54,302 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:54,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:54,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:54,304 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,311 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,311 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:54,311 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:54,311 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:54,311 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:54,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:54,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:54,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:54,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:54,311 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:54,311 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,311 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:54,311 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,312 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:54,312 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,312 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:54,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:54,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:54,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844094326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,407 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/0c17031f06a0456fabd22333b0424552 2023-04-18 18:53:54,412 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:54,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:54,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:54,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:54,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:54,416 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:54,416 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:54,416 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:54,416 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:54,417 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,417 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:54,418 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:54,418 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:54,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:54,419 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,419 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:54,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:54,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:54,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:54,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:54,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:54,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:54,420 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:54,420 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:54,420 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:54,420 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:54,420 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:54,420 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:54,421 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:54,421 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:54,421 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:54,421 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:54,421 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:54,421 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,421 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:54,421 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:54,424 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:54,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:54,424 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:54,426 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,426 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:54,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:54,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:54,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:54,428 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/61ee47bb9a434e77b342f406fbb2ab12 2023-04-18 18:53:54,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:54,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27b993e0[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:54,432 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:54,432 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,435 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,435 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:54,435 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:54,435 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/2f08a54c9c4944fe951b8c09d202c3d0 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/2f08a54c9c4944fe951b8c09d202c3d0 2023-04-18 18:53:54,435 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:54,436 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,436 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:54,439 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:54,439 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,439 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:54,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:54,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:54,439 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:54,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:54,445 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/2f08a54c9c4944fe951b8c09d202c3d0, entries=200, sequenceid=466, filesize=14.3 K 2023-04-18 18:53:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,446 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/0c17031f06a0456fabd22333b0424552 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/0c17031f06a0456fabd22333b0424552 2023-04-18 18:53:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:54,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:54,446 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:54,447 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:54,447 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:54,447 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:54,447 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:54,449 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:54,449 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,449 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:54,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:54,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:54,449 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:54,452 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:54,452 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/0c17031f06a0456fabd22333b0424552, entries=150, sequenceid=466, filesize=11.9 K 2023-04-18 18:53:54,454 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/61ee47bb9a434e77b342f406fbb2ab12 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/61ee47bb9a434e77b342f406fbb2ab12 2023-04-18 18:53:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,458 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:54,458 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:54,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:54,458 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:54,458 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:54,458 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:54,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:54,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:54,459 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:54,459 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,459 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:54,459 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:54,459 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:54,459 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:54,459 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:54,461 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/61ee47bb9a434e77b342f406fbb2ab12, entries=150, sequenceid=466, filesize=11.9 K 2023-04-18 18:53:54,462 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 491ms, sequenceid=466, compaction requested=true 2023-04-18 18:53:54,462 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:54,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:54,462 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:53:54,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:53:54,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:53:54,462 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:53:54,463 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39612 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:53:54,463 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/A is initiating minor compaction (all files) 2023-04-18 18:53:54,463 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/A in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:54,464 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:53:54,464 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/3554697fb9704cfbb6574fb9a239bae6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/b3e68a88061746e89b66653d29fa38d2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/2f08a54c9c4944fe951b8c09d202c3d0] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=38.7 K 2023-04-18 18:53:54,464 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/B is initiating minor compaction (all files) 2023-04-18 18:53:54,464 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/B in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:54,464 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/51622548e0db47bfb5b0b6ef40d8b57a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/fd002fc60cd7419a8b666005babd5356, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/0c17031f06a0456fabd22333b0424552] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=36.3 K 2023-04-18 18:53:54,466 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 3554697fb9704cfbb6574fb9a239bae6, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1681844032096 2023-04-18 18:53:54,467 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 51622548e0db47bfb5b0b6ef40d8b57a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1681844032096 2023-04-18 18:53:54,467 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting b3e68a88061746e89b66653d29fa38d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681844032439 2023-04-18 18:53:54,467 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting fd002fc60cd7419a8b666005babd5356, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681844032439 2023-04-18 18:53:54,467 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 2f08a54c9c4944fe951b8c09d202c3d0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681844033613 2023-04-18 18:53:54,468 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 0c17031f06a0456fabd22333b0424552, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681844033613 2023-04-18 18:53:54,476 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#A#compaction#63 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:54,476 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#B#compaction#64 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:54,489 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/96495e41af0c479ab23ccce82cea661e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/96495e41af0c479ab23ccce82cea661e 2023-04-18 18:53:54,492 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/4b508abd9b3648b2b0dcb790e40b3cc6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/4b508abd9b3648b2b0dcb790e40b3cc6 2023-04-18 18:53:54,497 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/A of d1a51bf079ac6b54cf5bff45ab0a5c8e into 96495e41af0c479ab23ccce82cea661e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:54,497 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:54,497 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/A, priority=13, startTime=1681844034462; duration=0sec 2023-04-18 18:53:54,497 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:53:54,497 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:53:54,498 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:53:54,498 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/C is initiating minor compaction (all files) 2023-04-18 18:53:54,498 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/C in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:54,498 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/3eb57fcb666942d2a405772f88163607, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/5ee2eb17fa3046cba2f0a664e16e472e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/61ee47bb9a434e77b342f406fbb2ab12] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=36.3 K 2023-04-18 18:53:54,499 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 3eb57fcb666942d2a405772f88163607, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1681844032096 2023-04-18 18:53:54,499 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 5ee2eb17fa3046cba2f0a664e16e472e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681844032439 2023-04-18 18:53:54,500 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 61ee47bb9a434e77b342f406fbb2ab12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681844033613 2023-04-18 18:53:54,500 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/B of d1a51bf079ac6b54cf5bff45ab0a5c8e into 4b508abd9b3648b2b0dcb790e40b3cc6(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:54,500 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:54,500 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/B, priority=13, startTime=1681844034462; duration=0sec 2023-04-18 18:53:54,500 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:54,507 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#C#compaction#65 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:54,526 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/c869e83ccbd24751b97e50766543422b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/c869e83ccbd24751b97e50766543422b 2023-04-18 18:53:54,533 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/C of d1a51bf079ac6b54cf5bff45ab0a5c8e into c869e83ccbd24751b97e50766543422b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:54,533 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:54,533 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/C, priority=13, startTime=1681844034462; duration=0sec 2023-04-18 18:53:54,533 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:54,559 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:54,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:54,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:54,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:54,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:54,564 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:54,564 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:54,564 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:54,564 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:54,565 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,565 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:54,566 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,566 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:54,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:54,566 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,566 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:54,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:54,567 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:54,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:54,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:54,568 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:54,568 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:54,568 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:54,568 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:54,568 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:54,568 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:54,568 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:54,569 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 18:53:54,569 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:54,569 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:54,569 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:54,569 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:54,569 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:54,569 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:54,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:54,633 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:54,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:54,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844094677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:54,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844094782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:54,980 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/615f65f1bf304cae9a3c152b46fc3bb6 2023-04-18 18:53:54,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:54,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844094993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:55,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:55,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844095297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:55,426 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/5ec065ffbd0d4d80bd1e9bdd51368fdc 2023-04-18 18:53:55,442 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/0a0289bad7b84487b148b1c3fe44eff0 2023-04-18 18:53:55,448 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/615f65f1bf304cae9a3c152b46fc3bb6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/615f65f1bf304cae9a3c152b46fc3bb6 2023-04-18 18:53:55,454 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/615f65f1bf304cae9a3c152b46fc3bb6, entries=150, sequenceid=482, filesize=11.9 K 2023-04-18 18:53:55,456 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/5ec065ffbd0d4d80bd1e9bdd51368fdc as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/5ec065ffbd0d4d80bd1e9bdd51368fdc 2023-04-18 18:53:55,465 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/5ec065ffbd0d4d80bd1e9bdd51368fdc, entries=150, sequenceid=482, filesize=11.9 K 2023-04-18 18:53:55,466 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/0a0289bad7b84487b148b1c3fe44eff0 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/0a0289bad7b84487b148b1c3fe44eff0 2023-04-18 18:53:55,483 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/0a0289bad7b84487b148b1c3fe44eff0, entries=150, sequenceid=482, filesize=11.9 K 2023-04-18 18:53:55,485 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 917ms, sequenceid=482, compaction requested=false 2023-04-18 18:53:55,485 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:55,485 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:55,485 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:55,485 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:55,485 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:55,485 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:55,485 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:55,488 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:55,488 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:55,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:55,489 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:55,489 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:55,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:55,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:55,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:55,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:55,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:55,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d01bfc5[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:55,490 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:55,491 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:55,492 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:55,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:55,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:55,492 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:55,492 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:55,492 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:55,496 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:55,496 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:55,496 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:55,496 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:55,496 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:55,496 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:55,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:55,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:55,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:55,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:55,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:55,501 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:55,501 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:55,501 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:55,501 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:55,501 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:55,503 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:55,503 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:55,503 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:55,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:55,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:55,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:55,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:55,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:55,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:55,504 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:55,505 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:55,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:55,508 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:55,508 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:55,508 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:55,508 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:55,508 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:55,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:55,509 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:55,509 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:55,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:55,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:55,509 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:55,509 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:55,509 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:55,509 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:55,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:55,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:55,509 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:55,609 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:55,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:55,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:55,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:55,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:55,613 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:55,613 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:55,613 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:55,613 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:55,614 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:55,614 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:55,615 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:55,615 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:55,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:55,616 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:55,616 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:55,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:55,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:55,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:55,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:55,617 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:55,617 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:55,617 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:55,617 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:55,617 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:55,617 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:55,617 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:55,617 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-18 18:53:55,618 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:55,618 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:55,618 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:55,618 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:55,618 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:55,618 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:55,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:55,803 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:55,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:55,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844095829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:55,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:55,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844095931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,030 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/3e352168bb754e63b4f6c4a01057efb0 2023-04-18 18:53:56,086 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/77b9f0ddbb0c4ac9a9ebd8e97067a2be 2023-04-18 18:53:56,118 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/e36697df45614394af9259d51d4d1d4c 2023-04-18 18:53:56,134 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/3e352168bb754e63b4f6c4a01057efb0 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/3e352168bb754e63b4f6c4a01057efb0 2023-04-18 18:53:56,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:56,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844096136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,142 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/3e352168bb754e63b4f6c4a01057efb0, entries=150, sequenceid=501, filesize=11.9 K 2023-04-18 18:53:56,144 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/77b9f0ddbb0c4ac9a9ebd8e97067a2be as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/77b9f0ddbb0c4ac9a9ebd8e97067a2be 2023-04-18 18:53:56,153 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/77b9f0ddbb0c4ac9a9ebd8e97067a2be, entries=150, sequenceid=501, filesize=11.9 K 2023-04-18 18:53:56,154 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/e36697df45614394af9259d51d4d1d4c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/e36697df45614394af9259d51d4d1d4c 2023-04-18 18:53:56,181 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/e36697df45614394af9259d51d4d1d4c, entries=150, sequenceid=501, filesize=11.9 K 2023-04-18 18:53:56,184 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 566ms, sequenceid=501, compaction requested=true 2023-04-18 18:53:56,184 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:56,184 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:56,276 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:56,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:56,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:56,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:56,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:56,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:56,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76e4c794[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:56,280 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:56,280 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:56,337 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:56,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:56,338 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:56,338 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:56,338 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:56,338 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,338 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:56,400 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,400 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:56,400 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:56,400 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:56,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:56,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:56,407 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:56,407 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:56,407 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:56,407 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:56,407 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:56,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:56,444 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-18 18:53:56,444 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:56,445 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:56,445 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:56,445 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:56,445 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:56,445 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:56,461 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:56,461 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:56,461 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:56,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:56,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:56,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:56,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:56,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:56,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:56,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:56,464 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:56,467 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/0fe7b5cd8e9f485280f7f1656e8b6a3c 2023-04-18 18:53:56,467 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:56,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844096509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,619 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,620 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:56,620 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:56,620 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,620 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:56,620 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:56,620 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:56,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:56,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844096620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,621 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:56,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:56,622 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:56,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:56,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:56,622 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:56,622 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:56,622 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:56,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:56,722 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:56,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:56,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:56,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:56,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:56,736 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:56,736 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:56,736 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:56,736 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:56,737 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:56,737 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:56,784 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:56,784 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:56,784 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,784 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:56,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:56,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:56,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:56,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:56,786 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:56,786 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:56,786 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:56,787 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:56,787 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:56,787 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:56,787 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:56,787 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:56,787 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:56,787 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:56,787 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:56,787 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,787 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:56,787 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:56,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:56,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844096825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,846 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:56,846 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:56,846 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:56,847 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:56,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:56,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3da4250f[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:56,850 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:56,850 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:56,903 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/b03d980fa09348f7ac6982e1dc62cd5b 2023-04-18 18:53:56,915 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:56,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:56,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:56,915 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:56,915 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:56,915 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:56,916 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,916 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:56,968 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,968 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:56,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:56,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:56,969 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:56,969 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:56,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:56,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:56,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:56,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:56,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:56,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:56,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:56,972 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:56,972 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:56,972 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:56,972 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:56,972 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:57,046 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,046 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,046 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:57,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:57,046 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:57,048 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:57,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:57,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:57,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33426 deadline: 1681844097056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,059 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8214 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:53:57,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:57,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33468 deadline: 1681844097075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,077 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8183 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:53:57,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:57,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33446 deadline: 1681844097088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,090 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8196 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:53:57,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:57,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33454 deadline: 1681844097092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,095 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8233 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:53:57,107 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,107 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:57,107 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:57,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:57,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:57,107 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:57,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:57,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:57,107 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:57,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:57,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:57,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:57,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:57,108 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:57,108 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:57,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:57,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:57,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844097129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,208 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:57,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:57,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:57,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:57,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:57,212 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:57,212 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:57,213 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:57,213 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:57,213 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,213 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:57,245 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:57,245 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:57,246 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:57,246 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:57,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:57,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:57,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:57,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:57,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:57,247 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:57,247 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:57,247 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:57,247 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:57,247 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:57,247 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:57,247 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:57,248 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:57,248 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:57,248 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:57,248 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:57,248 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,248 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:57,248 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:57,275 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,275 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:57,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:57,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:57,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:57,280 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:57,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56e987c8[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:57,280 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:57,306 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:57,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:57,307 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,307 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:57,307 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:57,307 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:57,307 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:57,316 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/584a5dbb23d64b3c915e09259506604a 2023-04-18 18:53:57,323 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/0fe7b5cd8e9f485280f7f1656e8b6a3c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/0fe7b5cd8e9f485280f7f1656e8b6a3c 2023-04-18 18:53:57,329 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,329 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:57,330 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:57,330 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:57,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/0fe7b5cd8e9f485280f7f1656e8b6a3c, entries=200, sequenceid=515, filesize=14.3 K 2023-04-18 18:53:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/b03d980fa09348f7ac6982e1dc62cd5b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/b03d980fa09348f7ac6982e1dc62cd5b 2023-04-18 18:53:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:57,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:57,333 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:57,333 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:57,333 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:57,333 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:57,333 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:57,338 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/b03d980fa09348f7ac6982e1dc62cd5b, entries=150, sequenceid=515, filesize=11.9 K 2023-04-18 18:53:57,339 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/584a5dbb23d64b3c915e09259506604a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/584a5dbb23d64b3c915e09259506604a 2023-04-18 18:53:57,346 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/584a5dbb23d64b3c915e09259506604a, entries=150, sequenceid=515, filesize=11.9 K 2023-04-18 18:53:57,347 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 903ms, sequenceid=515, compaction requested=true 2023-04-18 18:53:57,347 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:57,347 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:57,347 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:53:57,347 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:53:57,347 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:53:57,347 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:53:57,349 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:53:57,349 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/B is initiating minor compaction (all files) 2023-04-18 18:53:57,350 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/B in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:57,350 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/4b508abd9b3648b2b0dcb790e40b3cc6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/5ec065ffbd0d4d80bd1e9bdd51368fdc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/77b9f0ddbb0c4ac9a9ebd8e97067a2be, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/b03d980fa09348f7ac6982e1dc62cd5b] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=48.3 K 2023-04-18 18:53:57,350 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:53:57,350 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 4b508abd9b3648b2b0dcb790e40b3cc6, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681844033613 2023-04-18 18:53:57,350 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/A is initiating minor compaction (all files) 2023-04-18 18:53:57,350 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/A in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:57,351 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/96495e41af0c479ab23ccce82cea661e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/615f65f1bf304cae9a3c152b46fc3bb6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/3e352168bb754e63b4f6c4a01057efb0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/0fe7b5cd8e9f485280f7f1656e8b6a3c] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=50.7 K 2023-04-18 18:53:57,351 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 5ec065ffbd0d4d80bd1e9bdd51368fdc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681844033984 2023-04-18 18:53:57,351 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 96495e41af0c479ab23ccce82cea661e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681844033613 2023-04-18 18:53:57,351 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 77b9f0ddbb0c4ac9a9ebd8e97067a2be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1681844034667 2023-04-18 18:53:57,352 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b03d980fa09348f7ac6982e1dc62cd5b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1681844035818 2023-04-18 18:53:57,352 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 615f65f1bf304cae9a3c152b46fc3bb6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681844033984 2023-04-18 18:53:57,352 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 3e352168bb754e63b4f6c4a01057efb0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1681844034667 2023-04-18 18:53:57,353 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 0fe7b5cd8e9f485280f7f1656e8b6a3c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1681844035818 2023-04-18 18:53:57,368 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,368 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:57,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:57,368 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:57,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,368 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#B#compaction#75 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:57,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:57,369 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,370 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#A#compaction#76 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:57,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:57,371 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:57,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:57,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,406 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,407 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:57,407 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:57,407 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,407 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:57,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:57,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:57,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:57,407 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:57,407 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:57,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:57,407 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:57,407 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:57,407 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:57,407 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:57,407 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:57,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:57,507 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:57,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:57,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:57,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:57,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:57,516 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:57,516 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:57,516 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:57,516 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:57,517 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,517 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:57,569 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:57,569 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:57,569 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:57,569 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:57,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:57,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:57,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:57,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:57,571 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:57,571 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:57,571 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:57,571 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:57,572 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:57,572 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:57,572 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:57,572 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-18 18:53:57,572 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:57,572 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:57,572 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:57,573 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:57,573 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:57,573 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:57,596 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=533 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/c94b29ba27ce4902ae89f3a3da524a1f 2023-04-18 18:53:57,611 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=533 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/047fc623eb4847b0b840b8da78b44ca8 2023-04-18 18:53:57,633 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=533 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/e9c62bbe3aa141758d2d32b0e624ae12 2023-04-18 18:53:57,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:57,635 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:57,640 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/c94b29ba27ce4902ae89f3a3da524a1f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/c94b29ba27ce4902ae89f3a3da524a1f 2023-04-18 18:53:57,650 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/c94b29ba27ce4902ae89f3a3da524a1f, entries=150, sequenceid=533, filesize=11.9 K 2023-04-18 18:53:57,651 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/047fc623eb4847b0b840b8da78b44ca8 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/047fc623eb4847b0b840b8da78b44ca8 2023-04-18 18:53:57,672 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/047fc623eb4847b0b840b8da78b44ca8, entries=150, sequenceid=533, filesize=11.9 K 2023-04-18 18:53:57,688 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/e9c62bbe3aa141758d2d32b0e624ae12 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/e9c62bbe3aa141758d2d32b0e624ae12 2023-04-18 18:53:57,696 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/e9c62bbe3aa141758d2d32b0e624ae12, entries=150, sequenceid=533, filesize=11.9 K 2023-04-18 18:53:57,697 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 125ms, sequenceid=533, compaction requested=true 2023-04-18 18:53:57,697 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:57,697 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:57,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:57,704 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-18 18:53:57,704 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:57,704 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:57,704 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:57,704 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:57,704 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:57,704 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:57,768 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,768 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:57,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:57,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:57,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:57,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:57,769 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:57,769 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:57,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844097763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:57,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:57,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4606f547[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:57,772 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:57,772 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:57,795 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/a59cf9d270164cc7bdb6834522f8db09 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/a59cf9d270164cc7bdb6834522f8db09 2023-04-18 18:53:57,799 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/41322224fe9942109a8d52fce2bfbbdf as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/41322224fe9942109a8d52fce2bfbbdf 2023-04-18 18:53:57,802 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/A of d1a51bf079ac6b54cf5bff45ab0a5c8e into a59cf9d270164cc7bdb6834522f8db09(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:57,802 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:57,802 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/A, priority=12, startTime=1681844037347; duration=0sec 2023-04-18 18:53:57,802 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:53:57,803 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:53:57,806 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:57,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:57,806 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:57,807 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,807 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:57,807 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61646 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:53:57,807 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/C is initiating minor compaction (all files) 2023-04-18 18:53:57,808 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/C in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:57,808 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/c869e83ccbd24751b97e50766543422b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/0a0289bad7b84487b148b1c3fe44eff0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/e36697df45614394af9259d51d4d1d4c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/584a5dbb23d64b3c915e09259506604a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/e9c62bbe3aa141758d2d32b0e624ae12] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=60.2 K 2023-04-18 18:53:57,809 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting c869e83ccbd24751b97e50766543422b, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681844033613 2023-04-18 18:53:57,809 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/B of d1a51bf079ac6b54cf5bff45ab0a5c8e into 41322224fe9942109a8d52fce2bfbbdf(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:57,809 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:57,809 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/B, priority=12, startTime=1681844037347; duration=0sec 2023-04-18 18:53:57,809 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:57,810 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 0a0289bad7b84487b148b1c3fe44eff0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681844033984 2023-04-18 18:53:57,811 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting e36697df45614394af9259d51d4d1d4c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1681844034667 2023-04-18 18:53:57,811 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 584a5dbb23d64b3c915e09259506604a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1681844035818 2023-04-18 18:53:57,812 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting e9c62bbe3aa141758d2d32b0e624ae12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=533, earliestPutTs=1681844036490 2023-04-18 18:53:57,845 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,845 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:57,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:57,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:57,845 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:57,845 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:57,846 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#C#compaction#81 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:57,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:57,849 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:57,849 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:57,849 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:57,849 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:57,849 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:57,872 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/578b7cb970dd4fb88f9b193222f127c5 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/578b7cb970dd4fb88f9b193222f127c5 2023-04-18 18:53:57,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:57,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844097871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,884 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/C of d1a51bf079ac6b54cf5bff45ab0a5c8e into 578b7cb970dd4fb88f9b193222f127c5(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:57,884 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:57,884 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/C, priority=11, startTime=1681844037347; duration=0sec 2023-04-18 18:53:57,885 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:57,891 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,891 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,891 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:57,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:57,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:57,892 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:57,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:57,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:57,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:57,894 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:57,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:57,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:57,983 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:57,983 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:57,983 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:57,984 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:57,984 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:57,984 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:57,984 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:57,984 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:57,983 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:57,984 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:57,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:57,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:57,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:57,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:57,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:57,984 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:57,985 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:58,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:58,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844098077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:58,085 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:58,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:58,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:58,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:58,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:58,091 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:58,091 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:58,091 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:58,091 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:58,091 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:58,091 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:58,126 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/e1f509a0aa284f459994b197e7987220 2023-04-18 18:53:58,159 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/b07d0c1d93e14320b8955f507d1d7209 2023-04-18 18:53:58,322 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:58,322 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:58,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:58,322 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:58,322 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:58,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:58,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:58,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:58,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:58,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:58,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:58,325 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:58,325 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:58,325 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:58,325 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:53:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:53:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:53:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:53:58,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:58,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844098382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:58,391 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:58,391 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:58,392 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:58,392 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:53:58,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:58,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:58,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:53:58,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@168e5274[Count = 0] remaining members to acquire global barrier 2023-04-18 18:53:58,396 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:53:58,396 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:58,475 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:58,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:58,476 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:53:58,476 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:58,476 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:53:58,476 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:53:58,476 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:53:58,547 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:58,547 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:53:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:58,547 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:58,547 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:53:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:58,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:53:58,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:53:58,552 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:53:58,552 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:53:58,552 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:53:58,552 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:53:58,552 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:53:58,614 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:58,614 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:58,614 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:58,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:53:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:53:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:58,615 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:58,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:58,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:58,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:53:58,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:58,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:53:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:58,616 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:53:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:53:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:58,674 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/b620cb0b705a4ebbb5567c93fb0171ba 2023-04-18 18:53:58,680 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/e1f509a0aa284f459994b197e7987220 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/e1f509a0aa284f459994b197e7987220 2023-04-18 18:53:58,684 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:58,684 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:58,684 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:58,684 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:58,684 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:58,684 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:58,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:53:58,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:58,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:53:58,684 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:53:58,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:53:58,685 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:53:58,685 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:53:58,684 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:53:58,684 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:53:58,685 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:58,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:53:58,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:53:58,686 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/e1f509a0aa284f459994b197e7987220, entries=200, sequenceid=548, filesize=14.3 K 2023-04-18 18:53:58,688 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/b07d0c1d93e14320b8955f507d1d7209 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/b07d0c1d93e14320b8955f507d1d7209 2023-04-18 18:53:58,693 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/b07d0c1d93e14320b8955f507d1d7209, entries=150, sequenceid=548, filesize=11.9 K 2023-04-18 18:53:58,694 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/b620cb0b705a4ebbb5567c93fb0171ba as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/b620cb0b705a4ebbb5567c93fb0171ba 2023-04-18 18:53:58,700 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/b620cb0b705a4ebbb5567c93fb0171ba, entries=150, sequenceid=548, filesize=11.9 K 2023-04-18 18:53:58,701 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 997ms, sequenceid=548, compaction requested=true 2023-04-18 18:53:58,701 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:58,701 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:58,701 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:53:58,702 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:53:58,701 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:53:58,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:53:58,703 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39850 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:53:58,703 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/A is initiating minor compaction (all files) 2023-04-18 18:53:58,703 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:53:58,703 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/A in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:58,703 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/B is initiating minor compaction (all files) 2023-04-18 18:53:58,703 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/a59cf9d270164cc7bdb6834522f8db09, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/c94b29ba27ce4902ae89f3a3da524a1f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/e1f509a0aa284f459994b197e7987220] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=38.9 K 2023-04-18 18:53:58,703 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/B in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:58,703 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/41322224fe9942109a8d52fce2bfbbdf, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/047fc623eb4847b0b840b8da78b44ca8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/b07d0c1d93e14320b8955f507d1d7209] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=36.5 K 2023-04-18 18:53:58,704 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting a59cf9d270164cc7bdb6834522f8db09, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1681844035818 2023-04-18 18:53:58,704 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 41322224fe9942109a8d52fce2bfbbdf, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1681844035818 2023-04-18 18:53:58,704 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting c94b29ba27ce4902ae89f3a3da524a1f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=533, earliestPutTs=1681844036490 2023-04-18 18:53:58,705 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 047fc623eb4847b0b840b8da78b44ca8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=533, earliestPutTs=1681844036490 2023-04-18 18:53:58,705 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b07d0c1d93e14320b8955f507d1d7209, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681844037670 2023-04-18 18:53:58,705 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting e1f509a0aa284f459994b197e7987220, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681844037655 2023-04-18 18:53:58,733 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#B#compaction#84 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:58,736 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#A#compaction#85 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:53:58,770 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/dbd27339d0174aae9f71400ce9d16d19 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/dbd27339d0174aae9f71400ce9d16d19 2023-04-18 18:53:58,778 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/B of d1a51bf079ac6b54cf5bff45ab0a5c8e into dbd27339d0174aae9f71400ce9d16d19(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:58,778 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:58,778 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/B, priority=13, startTime=1681844038701; duration=0sec 2023-04-18 18:53:58,779 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:53:58,779 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-18 18:53:58,780 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-18 18:53:58,780 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-18 18:53:58,780 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. because compaction request was cancelled 2023-04-18 18:53:58,785 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/aac3218a09084282932cbbf8b55d1da1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/aac3218a09084282932cbbf8b55d1da1 2023-04-18 18:53:58,785 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:53:58,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:53:58,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:53:58,794 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/A of d1a51bf079ac6b54cf5bff45ab0a5c8e into aac3218a09084282932cbbf8b55d1da1(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:53:58,794 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:53:58,794 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/A, priority=13, startTime=1681844038701; duration=0sec 2023-04-18 18:53:58,794 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:53:58,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:53:58,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:53:58,796 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:53:58,796 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:58,796 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:53:58,796 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:53:58,797 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:58,797 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:58,876 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:53:58,876 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:58,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:53:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:53:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:58,876 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:58,876 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:53:58,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:53:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:53:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:53:58,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:53:58,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:53:58,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:53:58,878 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:53:58,878 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:53:58,878 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:53:58,878 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:53:58,878 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:53:58,878 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:53:58,878 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:53:58,878 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-18 18:53:58,878 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:53:58,879 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:58,879 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:53:58,879 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:58,879 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:53:58,879 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:53:58,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:53:58,887 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:53:58,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:58,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844098923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:59,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:59,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844099028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:59,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:59,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844099231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:59,290 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=570 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/bb54a23bbfe741838da5b1b3cee5a2ad 2023-04-18 18:53:59,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:53:59,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844099535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:53:59,711 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=570 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/3cd8a835ff364b068ca0c6a4b386a444 2023-04-18 18:54:00,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:00,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844100040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:00,146 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=570 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/d57598297e7f474e984cdc5f5cffa582 2023-04-18 18:54:00,153 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/bb54a23bbfe741838da5b1b3cee5a2ad as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/bb54a23bbfe741838da5b1b3cee5a2ad 2023-04-18 18:54:00,158 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/bb54a23bbfe741838da5b1b3cee5a2ad, entries=150, sequenceid=570, filesize=11.9 K 2023-04-18 18:54:00,159 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/3cd8a835ff364b068ca0c6a4b386a444 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/3cd8a835ff364b068ca0c6a4b386a444 2023-04-18 18:54:00,167 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/3cd8a835ff364b068ca0c6a4b386a444, entries=150, sequenceid=570, filesize=11.9 K 2023-04-18 18:54:00,168 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/d57598297e7f474e984cdc5f5cffa582 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/d57598297e7f474e984cdc5f5cffa582 2023-04-18 18:54:00,173 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/d57598297e7f474e984cdc5f5cffa582, entries=150, sequenceid=570, filesize=11.9 K 2023-04-18 18:54:00,174 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 1296ms, sequenceid=570, compaction requested=true 2023-04-18 18:54:00,174 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:54:00,174 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:00,174 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:00,174 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:00,174 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:00,174 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:00,174 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:00,215 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:00,215 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:00,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:00,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:00,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:00,215 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:00,215 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:00,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:00,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:00,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:00,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:00,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:00,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:00,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a9e91f7[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:00,218 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:00,218 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:00,277 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:00,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:00,277 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:00,277 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:00,277 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:00,278 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:00,278 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:00,329 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:00,329 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:00,330 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:00,330 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:00,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:00,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:00,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:00,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:00,333 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:00,333 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:00,333 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:00,333 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:00,333 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:00,400 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:00,400 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:00,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:00,400 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:00,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:00,401 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:00,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:00,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:00,402 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:00,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:00,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:00,476 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:00,476 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:00,476 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:00,476 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:00,476 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:00,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:00,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:00,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:00,476 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:00,476 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:00,476 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:00,476 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:00,476 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:00,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:00,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:00,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:00,477 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:00,477 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:00,577 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:00,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:00,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:00,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:00,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:00,581 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:00,581 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:00,581 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:00,581 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:00,581 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:00,581 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:00,629 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:00,629 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:00,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:00,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:00,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:00,630 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:00,630 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:00,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:00,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:00,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:00,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:00,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:00,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:00,631 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:00,631 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:00,631 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:00,632 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:00,632 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:00,632 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:00,632 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:54:00,632 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-18 18:54:00,632 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:54:00,632 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:00,632 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:54:00,633 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:00,633 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:54:00,633 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:00,643 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=585 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/5c951977b2534c17a4d25854bdb64c63 2023-04-18 18:54:00,656 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=585 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/df4a612666af441d9daf3571fa1a1851 2023-04-18 18:54:01,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:54:01,046 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:54:01,075 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=585 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/29acfa4193824eb6b91160c32c22012a 2023-04-18 18:54:01,083 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/5c951977b2534c17a4d25854bdb64c63 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/5c951977b2534c17a4d25854bdb64c63 2023-04-18 18:54:01,094 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/5c951977b2534c17a4d25854bdb64c63, entries=150, sequenceid=585, filesize=11.9 K 2023-04-18 18:54:01,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:01,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844101093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,095 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/df4a612666af441d9daf3571fa1a1851 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/df4a612666af441d9daf3571fa1a1851 2023-04-18 18:54:01,116 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/df4a612666af441d9daf3571fa1a1851, entries=150, sequenceid=585, filesize=11.9 K 2023-04-18 18:54:01,118 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/29acfa4193824eb6b91160c32c22012a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/29acfa4193824eb6b91160c32c22012a 2023-04-18 18:54:01,125 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/29acfa4193824eb6b91160c32c22012a, entries=150, sequenceid=585, filesize=11.9 K 2023-04-18 18:54:01,126 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 494ms, sequenceid=585, compaction requested=true 2023-04-18 18:54:01,126 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:54:01,126 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:01,126 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:01,126 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:01,127 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,127 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:01,127 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:01,153 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:01,153 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:01,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:01,153 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:01,153 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:01,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:01,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:01,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:01,156 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:01,156 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:01,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10d380ab[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:01,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:54:01,197 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-18 18:54:01,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:54:01,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:01,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:54:01,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:01,198 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:54:01,198 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:01,211 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/274f12f1fca248e2889527ae479b59d7 2023-04-18 18:54:01,230 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:01,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:01,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:01,231 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,231 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:01,231 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:01,231 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:01,231 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:01,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:01,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 401 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844101241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,299 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,299 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:01,299 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:01,299 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:01,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:01,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:01,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:01,303 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:01,303 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:01,303 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:01,303 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:01,303 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:01,345 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:01,345 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:01,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:01,345 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:01,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:01,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:01,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:01,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:01,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:01,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:01,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:01,346 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:01,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:01,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:01,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:01,347 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:01,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:01,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 403 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844101345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,383 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:01,383 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:01,384 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:01,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:01,384 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:01,384 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:01,384 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:01,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:01,384 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,384 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:01,384 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:01,384 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:01,384 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:01,384 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:01,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:01,484 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:01,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:01,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:01,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:01,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:01,488 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:01,488 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:01,488 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:01,488 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:01,489 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:01,489 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:01,537 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:01,537 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:01,538 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,538 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:01,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:01,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:01,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:01,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:01,539 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:01,539 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:01,539 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:01,539 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:01,539 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:01,539 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:01,539 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:54:01,539 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:54:01,539 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:01,539 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:01,539 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:01,539 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,540 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:01,540 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:01,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:01,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 405 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844101550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,599 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,599 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:01,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:01,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:01,599 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:01,599 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:01,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:01,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:01,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@618ef33b[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:01,601 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:01,601 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:01,642 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/12e2832bf1374c77a53c87334b47bebe 2023-04-18 18:54:01,746 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:01,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:01,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:01,747 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:01,747 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:01,747 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,747 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:01,747 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:01,829 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,829 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:01,829 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:01,830 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:01,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:01,832 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:01,833 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:01,833 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:01,833 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:01,833 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:01,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:01,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 407 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844101856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,899 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:01,899 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:01,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:01,899 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:01,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:01,900 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:01,902 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,945 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,945 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:01,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:01,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:01,945 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:01,945 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:01,945 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:01,945 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:01,945 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:01,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:01,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:01,945 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:01,945 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:01,946 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:01,946 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:01,946 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:01,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:02,046 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:02,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:02,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:02,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:02,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:02,049 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:02,049 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:02,049 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:02,049 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:02,050 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:02,050 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:02,054 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/b1fa028aec3e4651bd348dba531ddedc 2023-04-18 18:54:02,060 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/274f12f1fca248e2889527ae479b59d7 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/274f12f1fca248e2889527ae479b59d7 2023-04-18 18:54:02,066 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/274f12f1fca248e2889527ae479b59d7, entries=200, sequenceid=603, filesize=14.3 K 2023-04-18 18:54:02,067 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/12e2832bf1374c77a53c87334b47bebe as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/12e2832bf1374c77a53c87334b47bebe 2023-04-18 18:54:02,072 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/12e2832bf1374c77a53c87334b47bebe, entries=150, sequenceid=603, filesize=11.9 K 2023-04-18 18:54:02,074 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/b1fa028aec3e4651bd348dba531ddedc as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/b1fa028aec3e4651bd348dba531ddedc 2023-04-18 18:54:02,081 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/b1fa028aec3e4651bd348dba531ddedc, entries=150, sequenceid=603, filesize=11.9 K 2023-04-18 18:54:02,082 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 885ms, sequenceid=603, compaction requested=true 2023-04-18 18:54:02,082 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:54:02,082 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:02,082 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:02,082 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:54:02,082 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:02,082 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:54:02,084 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52138 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:54:02,084 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/A is initiating minor compaction (all files) 2023-04-18 18:54:02,084 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/A in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:02,084 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/aac3218a09084282932cbbf8b55d1da1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/bb54a23bbfe741838da5b1b3cee5a2ad, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/5c951977b2534c17a4d25854bdb64c63, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/274f12f1fca248e2889527ae479b59d7] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=50.9 K 2023-04-18 18:54:02,085 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:54:02,085 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/B is initiating minor compaction (all files) 2023-04-18 18:54:02,085 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/B in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:02,085 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/dbd27339d0174aae9f71400ce9d16d19, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/3cd8a835ff364b068ca0c6a4b386a444, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/df4a612666af441d9daf3571fa1a1851, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/12e2832bf1374c77a53c87334b47bebe] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=48.5 K 2023-04-18 18:54:02,086 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting aac3218a09084282932cbbf8b55d1da1, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681844037670 2023-04-18 18:54:02,089 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting dbd27339d0174aae9f71400ce9d16d19, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681844037670 2023-04-18 18:54:02,089 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting bb54a23bbfe741838da5b1b3cee5a2ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=570, earliestPutTs=1681844037754 2023-04-18 18:54:02,089 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 3cd8a835ff364b068ca0c6a4b386a444, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=570, earliestPutTs=1681844037754 2023-04-18 18:54:02,092 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 5c951977b2534c17a4d25854bdb64c63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1681844038915 2023-04-18 18:54:02,092 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting df4a612666af441d9daf3571fa1a1851, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1681844038915 2023-04-18 18:54:02,093 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 274f12f1fca248e2889527ae479b59d7, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681844041074 2023-04-18 18:54:02,093 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 12e2832bf1374c77a53c87334b47bebe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681844041087 2023-04-18 18:54:02,106 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#A#compaction#95 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:02,109 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#B#compaction#96 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:02,137 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:02,137 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:02,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:02,138 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:02,138 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:02,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:02,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:02,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:02,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:02,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:02,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:02,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:02,140 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:02,140 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:02,140 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:02,144 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:02,144 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:02,144 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:02,144 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:54:02,144 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-18 18:54:02,144 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:54:02,144 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:02,144 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:54:02,144 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:02,144 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:54:02,144 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:02,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:54:02,366 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:54:02,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:02,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 424 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844102454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:02,544 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/c2ffcf8cf3e543fe8d98e4fedb3f2480 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/c2ffcf8cf3e543fe8d98e4fedb3f2480 2023-04-18 18:54:02,545 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/2a7c2ff58fee42978c5a9e25d5801f8b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/2a7c2ff58fee42978c5a9e25d5801f8b 2023-04-18 18:54:02,550 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/B of d1a51bf079ac6b54cf5bff45ab0a5c8e into c2ffcf8cf3e543fe8d98e4fedb3f2480(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:02,550 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/A of d1a51bf079ac6b54cf5bff45ab0a5c8e into 2a7c2ff58fee42978c5a9e25d5801f8b(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:02,551 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:54:02,551 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:54:02,551 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/B, priority=12, startTime=1681844042082; duration=0sec 2023-04-18 18:54:02,551 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/A, priority=12, startTime=1681844042082; duration=0sec 2023-04-18 18:54:02,551 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:02,551 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:54:02,551 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:02,553 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61816 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:54:02,553 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/C is initiating minor compaction (all files) 2023-04-18 18:54:02,553 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/C in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:02,553 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/578b7cb970dd4fb88f9b193222f127c5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/b620cb0b705a4ebbb5567c93fb0171ba, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/d57598297e7f474e984cdc5f5cffa582, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/29acfa4193824eb6b91160c32c22012a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/b1fa028aec3e4651bd348dba531ddedc] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=60.4 K 2023-04-18 18:54:02,553 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 578b7cb970dd4fb88f9b193222f127c5, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=533, earliestPutTs=1681844036490 2023-04-18 18:54:02,554 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b620cb0b705a4ebbb5567c93fb0171ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681844037670 2023-04-18 18:54:02,554 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting d57598297e7f474e984cdc5f5cffa582, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=570, earliestPutTs=1681844037754 2023-04-18 18:54:02,555 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 29acfa4193824eb6b91160c32c22012a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1681844038915 2023-04-18 18:54:02,555 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b1fa028aec3e4651bd348dba531ddedc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681844041087 2023-04-18 18:54:02,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:02,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 426 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844102558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:02,561 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/30b56c2a05154a9d985b9d754b901bb7 2023-04-18 18:54:02,565 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#C#compaction#98 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:02,576 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/08f969ec3b5e4a4ca8607bc41b93fc55 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/08f969ec3b5e4a4ca8607bc41b93fc55 2023-04-18 18:54:02,581 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/C of d1a51bf079ac6b54cf5bff45ab0a5c8e into 08f969ec3b5e4a4ca8607bc41b93fc55(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:02,581 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:54:02,581 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/C, priority=11, startTime=1681844042082; duration=0sec 2023-04-18 18:54:02,581 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:02,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:02,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 428 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844102761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:02,974 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/ff5ae5427d3b4daea232dd93aeed0ee2 2023-04-18 18:54:02,993 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/db1e0fb1c60f4291933d05a6cee0026b 2023-04-18 18:54:02,999 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/30b56c2a05154a9d985b9d754b901bb7 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/30b56c2a05154a9d985b9d754b901bb7 2023-04-18 18:54:03,004 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/30b56c2a05154a9d985b9d754b901bb7, entries=150, sequenceid=617, filesize=11.9 K 2023-04-18 18:54:03,005 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/ff5ae5427d3b4daea232dd93aeed0ee2 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/ff5ae5427d3b4daea232dd93aeed0ee2 2023-04-18 18:54:03,010 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/ff5ae5427d3b4daea232dd93aeed0ee2, entries=150, sequenceid=617, filesize=11.9 K 2023-04-18 18:54:03,012 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/db1e0fb1c60f4291933d05a6cee0026b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/db1e0fb1c60f4291933d05a6cee0026b 2023-04-18 18:54:03,017 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/db1e0fb1c60f4291933d05a6cee0026b, entries=150, sequenceid=617, filesize=11.9 K 2023-04-18 18:54:03,018 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 874ms, sequenceid=617, compaction requested=false 2023-04-18 18:54:03,018 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:54:03,018 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:03,018 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:03,019 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:03,019 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:03,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:03,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:54:03,079 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-18 18:54:03,079 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:54:03,079 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:03,079 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:54:03,079 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:03,079 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:54:03,079 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:03,084 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,084 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:03,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:03,084 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,084 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:03,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:03,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:03,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:03,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:03,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@577108c[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:03,086 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:03,086 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:03,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 441 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844103119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,139 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,139 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:03,139 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:03,139 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:03,140 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,140 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:03,191 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,191 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:03,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:03,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:03,191 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:03,191 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:03,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:03,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:03,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:03,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:03,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:03,195 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:03,195 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:03,195 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:03,195 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:03,195 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:03,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:03,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 443 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844103224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,229 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,229 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:03,230 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,230 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,232 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,283 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,283 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:03,283 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:03,283 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:03,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:03,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:03,283 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:03,283 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:03,284 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,284 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:03,284 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,284 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:03,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:03,284 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,284 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:03,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:03,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:03,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:03,384 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:03,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:03,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:03,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:03,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:03,389 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:03,389 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:03,389 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:03,389 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:03,390 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,390 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:03,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:03,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 445 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844103427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,437 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:03,437 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:03,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:03,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:03,438 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,438 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:03,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:03,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:03,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:03,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:03,439 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:03,439 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:03,439 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:03,439 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:03,440 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:03,440 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:03,440 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:54:03,440 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:54:03,440 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:03,440 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:03,440 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:03,440 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,440 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:03,440 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:03,476 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,476 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:03,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:03,476 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,476 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:03,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:03,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:03,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:03,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:03,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b9838fc[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:03,478 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:03,478 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,504 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=639 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/5e2b6ef799be46eea4d5d9d0e8e0e423 2023-04-18 18:54:03,545 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,545 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:03,545 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:03,545 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,545 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:03,545 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:03,568 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,568 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:03,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:03,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:03,568 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:03,569 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:03,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:03,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:03,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:03,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:03,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:03,571 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:03,571 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:03,571 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:03,571 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:03,571 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:03,584 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,584 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,585 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:03,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:03,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:03,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:03,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:03,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:03,588 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:03,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,590 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:03,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,599 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,599 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:03,600 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:03,600 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:03,600 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:03,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:03,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:03,600 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:03,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:03,601 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:03,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:03,601 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,601 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,601 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:03,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:03,601 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:03,601 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:03,701 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:03,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:03,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:03,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:03,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:03,705 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:03,705 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:03,705 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:03,705 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:03,705 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,705 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:03,707 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,707 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:03,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:03,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:03,707 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,707 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:03,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:03,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:03,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:03,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:03,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:03,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:03,709 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:03,709 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:03,709 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:03,709 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:03,709 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:03,709 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:03,709 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:54:03,709 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:54:03,709 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:03,709 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:03,709 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:03,709 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,709 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:03,709 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:03,712 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,712 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:03,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:03,713 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,713 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:03,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:03,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:03,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:03,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:03,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@774239fc[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:03,715 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:03,715 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,716 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,717 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,717 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:03,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,720 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:03,720 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:03,721 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:03,726 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,726 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:03,726 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:03,726 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:03,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:03,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:03,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:03,730 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:03,730 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:03,730 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:03,730 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:03,730 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:03,732 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,732 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:03,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,732 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:03,733 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:03,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,734 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:03,738 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 447 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33410 deadline: 1681844103736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,738 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:03,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:03,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:03,738 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:03,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:03,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:03,738 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:03,738 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:03,738 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:03,738 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:03,738 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:03,739 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,739 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,739 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,739 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:03,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:03,839 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:03,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:03,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:03,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:03,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:03,842 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:03,842 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:03,842 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:03,842 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:03,843 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,843 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:03,844 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:03,844 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:03,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:03,845 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,845 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:03,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:03,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:03,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:03,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:03,847 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:03,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:03,847 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:03,847 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:03,847 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:03,847 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:03,847 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:03,847 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:03,847 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:54:03,848 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:54:03,848 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:03,848 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:03,848 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:03,848 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,848 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:03,848 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:03,850 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,850 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:03,851 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,851 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:03,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:03,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61430b04[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:03,852 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:03,852 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,854 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,854 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:03,854 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,854 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:03,854 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:03,854 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:03,855 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,855 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:03,855 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:03,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:03,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:03,855 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:03,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:03,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:03,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:03,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:03,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:03,858 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:03,858 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:03,858 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:03,858 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:03,858 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:03,860 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,860 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:03,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:03,860 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:03,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:03,861 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:03,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:03,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:03,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,862 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,865 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:03,865 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:03,865 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:03,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:03,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:03,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:03,866 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:03,866 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:03,866 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:03,866 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,866 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,866 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:03,866 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,866 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:03,866 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:03,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:03,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:03,917 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=639 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/f14b941c5fc749e197dc014715ace695 2023-04-18 18:54:03,931 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=639 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/7666e0c9bbf942fda0d09b70168de656 2023-04-18 18:54:03,938 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/5e2b6ef799be46eea4d5d9d0e8e0e423 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/5e2b6ef799be46eea4d5d9d0e8e0e423 2023-04-18 18:54:03,948 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/5e2b6ef799be46eea4d5d9d0e8e0e423, entries=200, sequenceid=639, filesize=14.3 K 2023-04-18 18:54:03,949 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/f14b941c5fc749e197dc014715ace695 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/f14b941c5fc749e197dc014715ace695 2023-04-18 18:54:03,954 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/f14b941c5fc749e197dc014715ace695, entries=150, sequenceid=639, filesize=11.9 K 2023-04-18 18:54:03,955 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/7666e0c9bbf942fda0d09b70168de656 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/7666e0c9bbf942fda0d09b70168de656 2023-04-18 18:54:03,959 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/7666e0c9bbf942fda0d09b70168de656, entries=150, sequenceid=639, filesize=11.9 K 2023-04-18 18:54:03,960 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 881ms, sequenceid=639, compaction requested=true 2023-04-18 18:54:03,960 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:54:03,960 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:03,960 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:54:03,960 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:54:03,960 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:03,960 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:03,961 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40088 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:54:03,961 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/A is initiating minor compaction (all files) 2023-04-18 18:54:03,961 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/A in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:03,962 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/2a7c2ff58fee42978c5a9e25d5801f8b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/30b56c2a05154a9d985b9d754b901bb7, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/5e2b6ef799be46eea4d5d9d0e8e0e423] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=39.1 K 2023-04-18 18:54:03,962 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:54:03,962 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 2a7c2ff58fee42978c5a9e25d5801f8b, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681844041087 2023-04-18 18:54:03,962 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/B is initiating minor compaction (all files) 2023-04-18 18:54:03,962 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/B in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:03,962 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/c2ffcf8cf3e543fe8d98e4fedb3f2480, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/ff5ae5427d3b4daea232dd93aeed0ee2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/f14b941c5fc749e197dc014715ace695] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=36.8 K 2023-04-18 18:54:03,963 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 30b56c2a05154a9d985b9d754b901bb7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1681844041230 2023-04-18 18:54:03,963 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 5e2b6ef799be46eea4d5d9d0e8e0e423, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1681844042421 2023-04-18 18:54:03,964 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting c2ffcf8cf3e543fe8d98e4fedb3f2480, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681844041087 2023-04-18 18:54:03,964 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting ff5ae5427d3b4daea232dd93aeed0ee2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1681844041230 2023-04-18 18:54:03,965 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting f14b941c5fc749e197dc014715ace695, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1681844042421 2023-04-18 18:54:03,966 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:03,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:03,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:03,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:03,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:03,976 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:03,976 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:03,976 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:03,976 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:03,977 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,977 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:03,978 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:03,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:03,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:03,978 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:03,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:03,980 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#A#compaction#104 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:03,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:03,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:03,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:03,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:03,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:03,980 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:03,980 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:03,980 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:03,980 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:03,980 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:03,981 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:03,981 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:03,981 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:03,981 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. started... 2023-04-18 18:54:03,981 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 18:54:03,981 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:54:03,981 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:03,981 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:54:03,982 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:03,982 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:54:03,982 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:03,983 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#B#compaction#105 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:04,009 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ec2fae8 to 127.0.0.1:65091 2023-04-18 18:54:04,009 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:54:04,011 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5979620a to 127.0.0.1:65091 2023-04-18 18:54:04,011 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:54:04,012 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00375c9d to 127.0.0.1:65091 2023-04-18 18:54:04,012 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:54:04,014 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27685d29 to 127.0.0.1:65091 2023-04-18 18:54:04,014 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:54:04,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:54:04,240 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. as already flushing 2023-04-18 18:54:04,241 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bc6d729 to 127.0.0.1:65091 2023-04-18 18:54:04,241 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:54:04,404 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/b9c9122573d44cc7b5bbc01049dc3a54 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/b9c9122573d44cc7b5bbc01049dc3a54 2023-04-18 18:54:04,408 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/5355fb26324e48edb1b95179508d53a4 2023-04-18 18:54:04,410 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/A of d1a51bf079ac6b54cf5bff45ab0a5c8e into b9c9122573d44cc7b5bbc01049dc3a54(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:04,410 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:54:04,410 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/A, priority=13, startTime=1681844043960; duration=0sec 2023-04-18 18:54:04,410 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:04,410 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:54:04,411 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/f045afcaaddb44efaf868eb3d66baeb5 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/f045afcaaddb44efaf868eb3d66baeb5 2023-04-18 18:54:04,412 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:54:04,412 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): d1a51bf079ac6b54cf5bff45ab0a5c8e/C is initiating minor compaction (all files) 2023-04-18 18:54:04,412 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d1a51bf079ac6b54cf5bff45ab0a5c8e/C in TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:04,412 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/08f969ec3b5e4a4ca8607bc41b93fc55, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/db1e0fb1c60f4291933d05a6cee0026b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/7666e0c9bbf942fda0d09b70168de656] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp, totalSize=36.7 K 2023-04-18 18:54:04,413 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 08f969ec3b5e4a4ca8607bc41b93fc55, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681844041087 2023-04-18 18:54:04,414 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting db1e0fb1c60f4291933d05a6cee0026b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1681844041230 2023-04-18 18:54:04,415 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 7666e0c9bbf942fda0d09b70168de656, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1681844042421 2023-04-18 18:54:04,418 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/B of d1a51bf079ac6b54cf5bff45ab0a5c8e into f045afcaaddb44efaf868eb3d66baeb5(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:04,418 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:54:04,418 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/B, priority=13, startTime=1681844043960; duration=0sec 2023-04-18 18:54:04,418 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:04,421 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/def5ee35eb20469696a3bd7cc3a7ea7a 2023-04-18 18:54:04,422 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): d1a51bf079ac6b54cf5bff45ab0a5c8e#C#compaction#108 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:04,832 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/9c076bf147f44454a4ff226e660a7669 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/9c076bf147f44454a4ff226e660a7669 2023-04-18 18:54:04,836 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/8923364551bc4f9ea9df72522e03d973 2023-04-18 18:54:04,837 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d1a51bf079ac6b54cf5bff45ab0a5c8e/C of d1a51bf079ac6b54cf5bff45ab0a5c8e into 9c076bf147f44454a4ff226e660a7669(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:04,837 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:54:04,837 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e., storeName=d1a51bf079ac6b54cf5bff45ab0a5c8e/C, priority=13, startTime=1681844043960; duration=0sec 2023-04-18 18:54:04,837 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:04,841 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/5355fb26324e48edb1b95179508d53a4 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/5355fb26324e48edb1b95179508d53a4 2023-04-18 18:54:04,845 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/5355fb26324e48edb1b95179508d53a4, entries=150, sequenceid=652, filesize=11.9 K 2023-04-18 18:54:04,845 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/def5ee35eb20469696a3bd7cc3a7ea7a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/def5ee35eb20469696a3bd7cc3a7ea7a 2023-04-18 18:54:04,849 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/def5ee35eb20469696a3bd7cc3a7ea7a, entries=150, sequenceid=652, filesize=11.9 K 2023-04-18 18:54:04,850 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/8923364551bc4f9ea9df72522e03d973 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/8923364551bc4f9ea9df72522e03d973 2023-04-18 18:54:04,854 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/8923364551bc4f9ea9df72522e03d973, entries=150, sequenceid=652, filesize=11.9 K 2023-04-18 18:54:04,855 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=6.71 KB/6870 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 874ms, sequenceid=652, compaction requested=false 2023-04-18 18:54:04,855 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:54:04,855 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:04,855 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:04,855 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:04,855 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:04,855 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:04,855 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:04,857 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:04,857 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:04,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:04,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:04,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:04,858 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:04,858 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:04,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:04,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:04,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:04,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:04,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:04,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:04,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c8d84d0[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:04,859 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:04,859 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:04,861 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:04,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:04,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:04,862 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:04,862 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:04,862 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:04,862 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:04,862 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:04,863 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:04,863 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:04,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:04,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:04,863 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:04,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:04,863 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:04,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:04,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:04,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:04,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:04,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:04,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:04,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:04,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:04,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:04,865 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:04,865 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:04,865 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:04,865 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:04,865 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:04,867 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:04,867 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:04,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:04,867 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:04,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:04,867 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:04,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:04,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:04,868 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:04,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:04,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:04,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:04,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:04,870 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:04,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:04,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:04,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:04,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:04,871 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:04,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:04,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:04,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:04,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:04,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:04,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:04,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:04,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:04,871 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:04,871 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:04,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:04,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:04,971 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:04,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:07,131 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1710407d to 127.0.0.1:65091 2023-04-18 18:54:07,131 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:54:07,141 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7b52aa42 to 127.0.0.1:65091 2023-04-18 18:54:07,141 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:54:07,142 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x72ea4a65 to 127.0.0.1:65091 2023-04-18 18:54:07,142 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:54:07,182 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1645b6b0 to 127.0.0.1:65091 2023-04-18 18:54:07,182 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:54:07,182 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-18 18:54:07,182 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 70 2023-04-18 18:54:07,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 59 2023-04-18 18:54:07,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 59 2023-04-18 18:54:07,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 58 2023-04-18 18:54:07,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 303 2023-04-18 18:54:07,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-18 18:54:07,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(398): read 4279 2023-04-18 18:54:07,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(398): read 4208 2023-04-18 18:54:07,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-18 18:54:07,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(402): scanned 1848 2023-04-18 18:54:07,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(403): verified 5543 rows 2023-04-18 18:54:07,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(402): scanned 1858 2023-04-18 18:54:07,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(403): verified 5573 rows 2023-04-18 18:54:07,183 INFO [Listener at localhost/36951] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-18 18:54:07,183 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d4f988c to 127.0.0.1:65091 2023-04-18 18:54:07,183 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:54:07,188 INFO [Listener at localhost/36951] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-18 18:54:07,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-18 18:54:07,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-18 18:54:07,204 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844047204"}]},"ts":"1681844047204"} 2023-04-18 18:54:07,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-18 18:54:07,205 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-18 18:54:07,207 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-18 18:54:07,208 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d1a51bf079ac6b54cf5bff45ab0a5c8e, UNASSIGN}] 2023-04-18 18:54:07,210 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d1a51bf079ac6b54cf5bff45ab0a5c8e, UNASSIGN 2023-04-18 18:54:07,211 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=d1a51bf079ac6b54cf5bff45ab0a5c8e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:07,211 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844047211"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681844047211"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681844047211"}]},"ts":"1681844047211"} 2023-04-18 18:54:07,213 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952}] 2023-04-18 18:54:07,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-18 18:54:07,370 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:54:07,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d1a51bf079ac6b54cf5bff45ab0a5c8e, disabling compactions & flushes 2023-04-18 18:54:07,370 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:07,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:07,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. after waiting 0 ms 2023-04-18 18:54:07,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:07,370 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing d1a51bf079ac6b54cf5bff45ab0a5c8e 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-18 18:54:07,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=A 2023-04-18 18:54:07,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:07,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=B 2023-04-18 18:54:07,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:07,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d1a51bf079ac6b54cf5bff45ab0a5c8e, store=C 2023-04-18 18:54:07,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:07,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-18 18:54:07,779 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=663 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/9cb15849e5f94514a142f26ebbaeab9e 2023-04-18 18:54:07,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-18 18:54:08,191 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=663 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/25151472070f4f98a7c7252a9079d8db 2023-04-18 18:54:08,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-18 18:54:08,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=663 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/53bd954cd8d8489f9c3a07668bea999f 2023-04-18 18:54:08,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/A/9cb15849e5f94514a142f26ebbaeab9e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/9cb15849e5f94514a142f26ebbaeab9e 2023-04-18 18:54:08,611 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/9cb15849e5f94514a142f26ebbaeab9e, entries=100, sequenceid=663, filesize=9.5 K 2023-04-18 18:54:08,612 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/B/25151472070f4f98a7c7252a9079d8db as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/25151472070f4f98a7c7252a9079d8db 2023-04-18 18:54:08,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/25151472070f4f98a7c7252a9079d8db, entries=100, sequenceid=663, filesize=9.5 K 2023-04-18 18:54:08,618 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/.tmp/C/53bd954cd8d8489f9c3a07668bea999f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/53bd954cd8d8489f9c3a07668bea999f 2023-04-18 18:54:08,623 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/53bd954cd8d8489f9c3a07668bea999f, entries=100, sequenceid=663, filesize=9.5 K 2023-04-18 18:54:08,624 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for d1a51bf079ac6b54cf5bff45ab0a5c8e in 1254ms, sequenceid=663, compaction requested=true 2023-04-18 18:54:08,646 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/b7e9409cfd0d49b083546649cb4337f2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/c73adec37e4b416cb5d6bbb61efe1ddb, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/f84fe57054da426698203ecffa3f0597, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/1d16005b3db84664b77910e34fd407e8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/da06836ad8584a91a09d047350251a91, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/f7f96e3216b14b60b159ee0369fbdb64, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/598de5b3676a483c8f030b4aea18d444, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/322a856f015b4679adb4dec2656ba67c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/d17096b41b774bec8310286fceb4bd9a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/d870df2417ac47ff87673ca3eddb892d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/1f7c1572cdb14318b51c5ca39a2aa577, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/761196f8f8a8427e8cc6e679f18acef0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/fd52d78731a14e1ab81a8111b8c6930f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/6c39866de56647c0b16eefd0af916b97, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/0486197ad40444f8a9150a5175bba25a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/33c34b9c131b4e8bbb491b6d2b8df3c8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/33bc1a2bd9e049a999463b3b36eb61d3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/38bad8a3ed304cbab273421d6faa5c91, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/3554697fb9704cfbb6574fb9a239bae6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/b3e68a88061746e89b66653d29fa38d2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/2f08a54c9c4944fe951b8c09d202c3d0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/96495e41af0c479ab23ccce82cea661e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/615f65f1bf304cae9a3c152b46fc3bb6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/3e352168bb754e63b4f6c4a01057efb0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/0fe7b5cd8e9f485280f7f1656e8b6a3c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/a59cf9d270164cc7bdb6834522f8db09, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/c94b29ba27ce4902ae89f3a3da524a1f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/e1f509a0aa284f459994b197e7987220, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/aac3218a09084282932cbbf8b55d1da1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/bb54a23bbfe741838da5b1b3cee5a2ad, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/5c951977b2534c17a4d25854bdb64c63, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/274f12f1fca248e2889527ae479b59d7, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/2a7c2ff58fee42978c5a9e25d5801f8b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/30b56c2a05154a9d985b9d754b901bb7, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/5e2b6ef799be46eea4d5d9d0e8e0e423] to archive 2023-04-18 18:54:08,647 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 18:54:08,654 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/b7e9409cfd0d49b083546649cb4337f2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/b7e9409cfd0d49b083546649cb4337f2 2023-04-18 18:54:08,656 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/c73adec37e4b416cb5d6bbb61efe1ddb to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/c73adec37e4b416cb5d6bbb61efe1ddb 2023-04-18 18:54:08,657 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/f84fe57054da426698203ecffa3f0597 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/f84fe57054da426698203ecffa3f0597 2023-04-18 18:54:08,659 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/1d16005b3db84664b77910e34fd407e8 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/1d16005b3db84664b77910e34fd407e8 2023-04-18 18:54:08,660 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/da06836ad8584a91a09d047350251a91 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/da06836ad8584a91a09d047350251a91 2023-04-18 18:54:08,662 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/f7f96e3216b14b60b159ee0369fbdb64 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/f7f96e3216b14b60b159ee0369fbdb64 2023-04-18 18:54:08,663 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/598de5b3676a483c8f030b4aea18d444 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/598de5b3676a483c8f030b4aea18d444 2023-04-18 18:54:08,665 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/322a856f015b4679adb4dec2656ba67c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/322a856f015b4679adb4dec2656ba67c 2023-04-18 18:54:08,666 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/d17096b41b774bec8310286fceb4bd9a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/d17096b41b774bec8310286fceb4bd9a 2023-04-18 18:54:08,667 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/d870df2417ac47ff87673ca3eddb892d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/d870df2417ac47ff87673ca3eddb892d 2023-04-18 18:54:08,668 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/1f7c1572cdb14318b51c5ca39a2aa577 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/1f7c1572cdb14318b51c5ca39a2aa577 2023-04-18 18:54:08,669 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/761196f8f8a8427e8cc6e679f18acef0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/761196f8f8a8427e8cc6e679f18acef0 2023-04-18 18:54:08,670 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/fd52d78731a14e1ab81a8111b8c6930f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/fd52d78731a14e1ab81a8111b8c6930f 2023-04-18 18:54:08,671 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/6c39866de56647c0b16eefd0af916b97 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/6c39866de56647c0b16eefd0af916b97 2023-04-18 18:54:08,672 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/0486197ad40444f8a9150a5175bba25a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/0486197ad40444f8a9150a5175bba25a 2023-04-18 18:54:08,674 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/33c34b9c131b4e8bbb491b6d2b8df3c8 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/33c34b9c131b4e8bbb491b6d2b8df3c8 2023-04-18 18:54:08,675 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/33bc1a2bd9e049a999463b3b36eb61d3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/33bc1a2bd9e049a999463b3b36eb61d3 2023-04-18 18:54:08,675 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/38bad8a3ed304cbab273421d6faa5c91 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/38bad8a3ed304cbab273421d6faa5c91 2023-04-18 18:54:08,677 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/3554697fb9704cfbb6574fb9a239bae6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/3554697fb9704cfbb6574fb9a239bae6 2023-04-18 18:54:08,678 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/b3e68a88061746e89b66653d29fa38d2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/b3e68a88061746e89b66653d29fa38d2 2023-04-18 18:54:08,679 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/2f08a54c9c4944fe951b8c09d202c3d0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/2f08a54c9c4944fe951b8c09d202c3d0 2023-04-18 18:54:08,680 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/96495e41af0c479ab23ccce82cea661e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/96495e41af0c479ab23ccce82cea661e 2023-04-18 18:54:08,681 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/615f65f1bf304cae9a3c152b46fc3bb6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/615f65f1bf304cae9a3c152b46fc3bb6 2023-04-18 18:54:08,682 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/3e352168bb754e63b4f6c4a01057efb0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/3e352168bb754e63b4f6c4a01057efb0 2023-04-18 18:54:08,683 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/0fe7b5cd8e9f485280f7f1656e8b6a3c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/0fe7b5cd8e9f485280f7f1656e8b6a3c 2023-04-18 18:54:08,684 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/a59cf9d270164cc7bdb6834522f8db09 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/a59cf9d270164cc7bdb6834522f8db09 2023-04-18 18:54:08,685 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/c94b29ba27ce4902ae89f3a3da524a1f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/c94b29ba27ce4902ae89f3a3da524a1f 2023-04-18 18:54:08,686 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/e1f509a0aa284f459994b197e7987220 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/e1f509a0aa284f459994b197e7987220 2023-04-18 18:54:08,687 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/aac3218a09084282932cbbf8b55d1da1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/aac3218a09084282932cbbf8b55d1da1 2023-04-18 18:54:08,688 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/bb54a23bbfe741838da5b1b3cee5a2ad to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/bb54a23bbfe741838da5b1b3cee5a2ad 2023-04-18 18:54:08,689 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/5c951977b2534c17a4d25854bdb64c63 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/5c951977b2534c17a4d25854bdb64c63 2023-04-18 18:54:08,690 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/274f12f1fca248e2889527ae479b59d7 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/274f12f1fca248e2889527ae479b59d7 2023-04-18 18:54:08,691 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/2a7c2ff58fee42978c5a9e25d5801f8b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/2a7c2ff58fee42978c5a9e25d5801f8b 2023-04-18 18:54:08,692 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/30b56c2a05154a9d985b9d754b901bb7 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/30b56c2a05154a9d985b9d754b901bb7 2023-04-18 18:54:08,693 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/5e2b6ef799be46eea4d5d9d0e8e0e423 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/5e2b6ef799be46eea4d5d9d0e8e0e423 2023-04-18 18:54:08,717 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/64315ef840d64b1bb93f1b38b59710ac, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/a81c73b161464bbcaf9e583bc709ade0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/65cb3f84cdd74fcfa65e6a31392cba2c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/189ac20ad72a4873a739d175b6e85809, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/862fbf81283d4c70ae6de7d87951bf89, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/42474ee0cd554a808f06705dd745d4fb, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/3bf17ebc97244987b12878fe3a4c47e9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/f6130a44d1fc473db0271a112631bebd, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/9b54115a30e446458be790e24bfe7d17, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/a5fe6a1ea6a940fda0e76d95f8604b9b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/d8a9513073724a0abd9281335f861f53, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/c149a5721f2e4127938aee1fe405fd24, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/65dfaf169bf54317918f577dffb63cbb, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/4f8103a65da84e05995d0ee6f89ba63a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/04ba904878e84ccbbe908f53b1172ef3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/17a9763e5afd49ef8029a260ebce8c9c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/ec2b7ecee7824dea8be53596f821db23, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/51622548e0db47bfb5b0b6ef40d8b57a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/4a1d8f207b8746d49e9f4fd0aedf5e60, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/fd002fc60cd7419a8b666005babd5356, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/4b508abd9b3648b2b0dcb790e40b3cc6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/0c17031f06a0456fabd22333b0424552, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/5ec065ffbd0d4d80bd1e9bdd51368fdc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/77b9f0ddbb0c4ac9a9ebd8e97067a2be, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/41322224fe9942109a8d52fce2bfbbdf, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/b03d980fa09348f7ac6982e1dc62cd5b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/047fc623eb4847b0b840b8da78b44ca8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/dbd27339d0174aae9f71400ce9d16d19, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/b07d0c1d93e14320b8955f507d1d7209, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/3cd8a835ff364b068ca0c6a4b386a444, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/df4a612666af441d9daf3571fa1a1851, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/c2ffcf8cf3e543fe8d98e4fedb3f2480, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/12e2832bf1374c77a53c87334b47bebe, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/ff5ae5427d3b4daea232dd93aeed0ee2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/f14b941c5fc749e197dc014715ace695] to archive 2023-04-18 18:54:08,718 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 18:54:08,720 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/64315ef840d64b1bb93f1b38b59710ac to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/64315ef840d64b1bb93f1b38b59710ac 2023-04-18 18:54:08,721 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/a81c73b161464bbcaf9e583bc709ade0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/a81c73b161464bbcaf9e583bc709ade0 2023-04-18 18:54:08,722 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/65cb3f84cdd74fcfa65e6a31392cba2c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/65cb3f84cdd74fcfa65e6a31392cba2c 2023-04-18 18:54:08,723 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/189ac20ad72a4873a739d175b6e85809 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/189ac20ad72a4873a739d175b6e85809 2023-04-18 18:54:08,724 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/862fbf81283d4c70ae6de7d87951bf89 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/862fbf81283d4c70ae6de7d87951bf89 2023-04-18 18:54:08,725 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/42474ee0cd554a808f06705dd745d4fb to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/42474ee0cd554a808f06705dd745d4fb 2023-04-18 18:54:08,726 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/3bf17ebc97244987b12878fe3a4c47e9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/3bf17ebc97244987b12878fe3a4c47e9 2023-04-18 18:54:08,727 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/f6130a44d1fc473db0271a112631bebd to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/f6130a44d1fc473db0271a112631bebd 2023-04-18 18:54:08,728 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/9b54115a30e446458be790e24bfe7d17 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/9b54115a30e446458be790e24bfe7d17 2023-04-18 18:54:08,729 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/a5fe6a1ea6a940fda0e76d95f8604b9b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/a5fe6a1ea6a940fda0e76d95f8604b9b 2023-04-18 18:54:08,730 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/d8a9513073724a0abd9281335f861f53 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/d8a9513073724a0abd9281335f861f53 2023-04-18 18:54:08,731 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/c149a5721f2e4127938aee1fe405fd24 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/c149a5721f2e4127938aee1fe405fd24 2023-04-18 18:54:08,733 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/65dfaf169bf54317918f577dffb63cbb to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/65dfaf169bf54317918f577dffb63cbb 2023-04-18 18:54:08,734 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/4f8103a65da84e05995d0ee6f89ba63a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/4f8103a65da84e05995d0ee6f89ba63a 2023-04-18 18:54:08,735 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/04ba904878e84ccbbe908f53b1172ef3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/04ba904878e84ccbbe908f53b1172ef3 2023-04-18 18:54:08,736 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/17a9763e5afd49ef8029a260ebce8c9c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/17a9763e5afd49ef8029a260ebce8c9c 2023-04-18 18:54:08,737 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/ec2b7ecee7824dea8be53596f821db23 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/ec2b7ecee7824dea8be53596f821db23 2023-04-18 18:54:08,738 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/51622548e0db47bfb5b0b6ef40d8b57a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/51622548e0db47bfb5b0b6ef40d8b57a 2023-04-18 18:54:08,739 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/4a1d8f207b8746d49e9f4fd0aedf5e60 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/4a1d8f207b8746d49e9f4fd0aedf5e60 2023-04-18 18:54:08,740 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/fd002fc60cd7419a8b666005babd5356 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/fd002fc60cd7419a8b666005babd5356 2023-04-18 18:54:08,741 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/4b508abd9b3648b2b0dcb790e40b3cc6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/4b508abd9b3648b2b0dcb790e40b3cc6 2023-04-18 18:54:08,742 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/0c17031f06a0456fabd22333b0424552 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/0c17031f06a0456fabd22333b0424552 2023-04-18 18:54:08,744 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/5ec065ffbd0d4d80bd1e9bdd51368fdc to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/5ec065ffbd0d4d80bd1e9bdd51368fdc 2023-04-18 18:54:08,745 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/77b9f0ddbb0c4ac9a9ebd8e97067a2be to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/77b9f0ddbb0c4ac9a9ebd8e97067a2be 2023-04-18 18:54:08,746 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/41322224fe9942109a8d52fce2bfbbdf to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/41322224fe9942109a8d52fce2bfbbdf 2023-04-18 18:54:08,747 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/b03d980fa09348f7ac6982e1dc62cd5b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/b03d980fa09348f7ac6982e1dc62cd5b 2023-04-18 18:54:08,748 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/047fc623eb4847b0b840b8da78b44ca8 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/047fc623eb4847b0b840b8da78b44ca8 2023-04-18 18:54:08,749 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/dbd27339d0174aae9f71400ce9d16d19 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/dbd27339d0174aae9f71400ce9d16d19 2023-04-18 18:54:08,750 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/b07d0c1d93e14320b8955f507d1d7209 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/b07d0c1d93e14320b8955f507d1d7209 2023-04-18 18:54:08,751 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/3cd8a835ff364b068ca0c6a4b386a444 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/3cd8a835ff364b068ca0c6a4b386a444 2023-04-18 18:54:08,752 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/df4a612666af441d9daf3571fa1a1851 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/df4a612666af441d9daf3571fa1a1851 2023-04-18 18:54:08,754 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/c2ffcf8cf3e543fe8d98e4fedb3f2480 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/c2ffcf8cf3e543fe8d98e4fedb3f2480 2023-04-18 18:54:08,755 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/12e2832bf1374c77a53c87334b47bebe to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/12e2832bf1374c77a53c87334b47bebe 2023-04-18 18:54:08,756 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/ff5ae5427d3b4daea232dd93aeed0ee2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/ff5ae5427d3b4daea232dd93aeed0ee2 2023-04-18 18:54:08,757 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/f14b941c5fc749e197dc014715ace695 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/f14b941c5fc749e197dc014715ace695 2023-04-18 18:54:08,761 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/4d77308ebb3048b694f07a26e7cbf651, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/3d87554627f141e093bcaf68d5bdd9c5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/aaa88160a4fa416492fed07aabeba119, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/fbd61286b90946398d594e64bb17d43f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/b01a6603515e4d4d949c66c67cece9f5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/910a494e285645e9aa67a8a70e2c37d2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/61122ce2190c44a998b9f6c720f7358d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/1f6aac2d31204612899b019afcfd2b5c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/9b24d62a4494401fa1ded32be9a3a43f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/7e35225f1c5d460d9e8b0662c2f2aada, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/ab81b78a68e24cc69d69bbd30efab104, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/2bb995d1cb164024ad87cde65e8d5430, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/46a9fbc8185946f3a3fc51a382ecbc3a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/f1ee4e644cb84232a44db82a6ab71450, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/35cff34ae4594e9c884864b85b999bd6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/afb04eaea7bb453e9b1743e7534dff80, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/71d72e0f42854b5c81df660f1b7d9a92, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/3eb57fcb666942d2a405772f88163607, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/1cf4f303a2fd4a78ab93f0a98cbea1e3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/5ee2eb17fa3046cba2f0a664e16e472e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/c869e83ccbd24751b97e50766543422b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/61ee47bb9a434e77b342f406fbb2ab12, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/0a0289bad7b84487b148b1c3fe44eff0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/e36697df45614394af9259d51d4d1d4c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/584a5dbb23d64b3c915e09259506604a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/578b7cb970dd4fb88f9b193222f127c5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/e9c62bbe3aa141758d2d32b0e624ae12, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/b620cb0b705a4ebbb5567c93fb0171ba, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/d57598297e7f474e984cdc5f5cffa582, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/29acfa4193824eb6b91160c32c22012a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/08f969ec3b5e4a4ca8607bc41b93fc55, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/b1fa028aec3e4651bd348dba531ddedc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/db1e0fb1c60f4291933d05a6cee0026b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/7666e0c9bbf942fda0d09b70168de656] to archive 2023-04-18 18:54:08,762 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 18:54:08,764 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/4d77308ebb3048b694f07a26e7cbf651 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/4d77308ebb3048b694f07a26e7cbf651 2023-04-18 18:54:08,766 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/3d87554627f141e093bcaf68d5bdd9c5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/3d87554627f141e093bcaf68d5bdd9c5 2023-04-18 18:54:08,767 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/aaa88160a4fa416492fed07aabeba119 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/aaa88160a4fa416492fed07aabeba119 2023-04-18 18:54:08,769 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/fbd61286b90946398d594e64bb17d43f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/fbd61286b90946398d594e64bb17d43f 2023-04-18 18:54:08,770 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/b01a6603515e4d4d949c66c67cece9f5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/b01a6603515e4d4d949c66c67cece9f5 2023-04-18 18:54:08,771 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/910a494e285645e9aa67a8a70e2c37d2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/910a494e285645e9aa67a8a70e2c37d2 2023-04-18 18:54:08,772 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/61122ce2190c44a998b9f6c720f7358d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/61122ce2190c44a998b9f6c720f7358d 2023-04-18 18:54:08,773 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/1f6aac2d31204612899b019afcfd2b5c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/1f6aac2d31204612899b019afcfd2b5c 2023-04-18 18:54:08,775 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/9b24d62a4494401fa1ded32be9a3a43f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/9b24d62a4494401fa1ded32be9a3a43f 2023-04-18 18:54:08,776 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/7e35225f1c5d460d9e8b0662c2f2aada to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/7e35225f1c5d460d9e8b0662c2f2aada 2023-04-18 18:54:08,777 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/ab81b78a68e24cc69d69bbd30efab104 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/ab81b78a68e24cc69d69bbd30efab104 2023-04-18 18:54:08,778 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/2bb995d1cb164024ad87cde65e8d5430 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/2bb995d1cb164024ad87cde65e8d5430 2023-04-18 18:54:08,779 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/46a9fbc8185946f3a3fc51a382ecbc3a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/46a9fbc8185946f3a3fc51a382ecbc3a 2023-04-18 18:54:08,780 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/f1ee4e644cb84232a44db82a6ab71450 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/f1ee4e644cb84232a44db82a6ab71450 2023-04-18 18:54:08,781 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/35cff34ae4594e9c884864b85b999bd6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/35cff34ae4594e9c884864b85b999bd6 2023-04-18 18:54:08,783 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/afb04eaea7bb453e9b1743e7534dff80 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/afb04eaea7bb453e9b1743e7534dff80 2023-04-18 18:54:08,784 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/71d72e0f42854b5c81df660f1b7d9a92 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/71d72e0f42854b5c81df660f1b7d9a92 2023-04-18 18:54:08,785 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/3eb57fcb666942d2a405772f88163607 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/3eb57fcb666942d2a405772f88163607 2023-04-18 18:54:08,786 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/1cf4f303a2fd4a78ab93f0a98cbea1e3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/1cf4f303a2fd4a78ab93f0a98cbea1e3 2023-04-18 18:54:08,787 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/5ee2eb17fa3046cba2f0a664e16e472e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/5ee2eb17fa3046cba2f0a664e16e472e 2023-04-18 18:54:08,788 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/c869e83ccbd24751b97e50766543422b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/c869e83ccbd24751b97e50766543422b 2023-04-18 18:54:08,789 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/61ee47bb9a434e77b342f406fbb2ab12 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/61ee47bb9a434e77b342f406fbb2ab12 2023-04-18 18:54:08,790 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/0a0289bad7b84487b148b1c3fe44eff0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/0a0289bad7b84487b148b1c3fe44eff0 2023-04-18 18:54:08,792 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/e36697df45614394af9259d51d4d1d4c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/e36697df45614394af9259d51d4d1d4c 2023-04-18 18:54:08,793 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/584a5dbb23d64b3c915e09259506604a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/584a5dbb23d64b3c915e09259506604a 2023-04-18 18:54:08,794 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/578b7cb970dd4fb88f9b193222f127c5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/578b7cb970dd4fb88f9b193222f127c5 2023-04-18 18:54:08,795 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/e9c62bbe3aa141758d2d32b0e624ae12 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/e9c62bbe3aa141758d2d32b0e624ae12 2023-04-18 18:54:08,796 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/b620cb0b705a4ebbb5567c93fb0171ba to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/b620cb0b705a4ebbb5567c93fb0171ba 2023-04-18 18:54:08,797 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/d57598297e7f474e984cdc5f5cffa582 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/d57598297e7f474e984cdc5f5cffa582 2023-04-18 18:54:08,799 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/29acfa4193824eb6b91160c32c22012a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/29acfa4193824eb6b91160c32c22012a 2023-04-18 18:54:08,800 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/08f969ec3b5e4a4ca8607bc41b93fc55 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/08f969ec3b5e4a4ca8607bc41b93fc55 2023-04-18 18:54:08,801 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/b1fa028aec3e4651bd348dba531ddedc to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/b1fa028aec3e4651bd348dba531ddedc 2023-04-18 18:54:08,802 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/db1e0fb1c60f4291933d05a6cee0026b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/db1e0fb1c60f4291933d05a6cee0026b 2023-04-18 18:54:08,803 DEBUG [StoreCloser-TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/7666e0c9bbf942fda0d09b70168de656 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/7666e0c9bbf942fda0d09b70168de656 2023-04-18 18:54:08,808 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/recovered.edits/666.seqid, newMaxSeqId=666, maxSeqId=1 2023-04-18 18:54:08,810 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e. 2023-04-18 18:54:08,810 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d1a51bf079ac6b54cf5bff45ab0a5c8e: 2023-04-18 18:54:08,812 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:54:08,813 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=d1a51bf079ac6b54cf5bff45ab0a5c8e, regionState=CLOSED 2023-04-18 18:54:08,813 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844048813"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681844048813"}]},"ts":"1681844048813"} 2023-04-18 18:54:08,816 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-18 18:54:08,816 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure d1a51bf079ac6b54cf5bff45ab0a5c8e, server=jenkins-hbase4.apache.org,34053,1681844018952 in 1.6020 sec 2023-04-18 18:54:08,817 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-18 18:54:08,817 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d1a51bf079ac6b54cf5bff45ab0a5c8e, UNASSIGN in 1.6080 sec 2023-04-18 18:54:08,818 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844048818"}]},"ts":"1681844048818"} 2023-04-18 18:54:08,819 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-18 18:54:08,821 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-18 18:54:08,823 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6270 sec 2023-04-18 18:54:09,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-18 18:54:09,309 INFO [Listener at localhost/36951] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-18 18:54:09,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-18 18:54:09,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:54:09,321 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:54:09,323 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:54:09,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-18 18:54:09,328 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:54:09,331 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A, FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B, FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C, FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/recovered.edits] 2023-04-18 18:54:09,334 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/5355fb26324e48edb1b95179508d53a4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/5355fb26324e48edb1b95179508d53a4 2023-04-18 18:54:09,335 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/9cb15849e5f94514a142f26ebbaeab9e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/9cb15849e5f94514a142f26ebbaeab9e 2023-04-18 18:54:09,336 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/b9c9122573d44cc7b5bbc01049dc3a54 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/A/b9c9122573d44cc7b5bbc01049dc3a54 2023-04-18 18:54:09,338 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/25151472070f4f98a7c7252a9079d8db to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/25151472070f4f98a7c7252a9079d8db 2023-04-18 18:54:09,339 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/def5ee35eb20469696a3bd7cc3a7ea7a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/def5ee35eb20469696a3bd7cc3a7ea7a 2023-04-18 18:54:09,340 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/f045afcaaddb44efaf868eb3d66baeb5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/B/f045afcaaddb44efaf868eb3d66baeb5 2023-04-18 18:54:09,342 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/53bd954cd8d8489f9c3a07668bea999f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/53bd954cd8d8489f9c3a07668bea999f 2023-04-18 18:54:09,343 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/8923364551bc4f9ea9df72522e03d973 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/8923364551bc4f9ea9df72522e03d973 2023-04-18 18:54:09,345 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/9c076bf147f44454a4ff226e660a7669 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/C/9c076bf147f44454a4ff226e660a7669 2023-04-18 18:54:09,347 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/recovered.edits/666.seqid to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e/recovered.edits/666.seqid 2023-04-18 18:54:09,348 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/d1a51bf079ac6b54cf5bff45ab0a5c8e 2023-04-18 18:54:09,348 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 18:54:09,351 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:54:09,372 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-18 18:54:09,416 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-18 18:54:09,418 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:54:09,418 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-18 18:54:09,418 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681844049418"}]},"ts":"9223372036854775807"} 2023-04-18 18:54:09,420 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-18 18:54:09,420 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d1a51bf079ac6b54cf5bff45ab0a5c8e, NAME => 'TestAcidGuarantees,,1681844022706.d1a51bf079ac6b54cf5bff45ab0a5c8e.', STARTKEY => '', ENDKEY => ''}] 2023-04-18 18:54:09,420 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-18 18:54:09,421 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681844049421"}]},"ts":"9223372036854775807"} 2023-04-18 18:54:09,423 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-18 18:54:09,425 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:54:09,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-18 18:54:09,426 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 110 msec 2023-04-18 18:54:09,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-18 18:54:09,627 INFO [Listener at localhost/36951] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-18 18:54:09,644 INFO [Listener at localhost/36951] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=235 (was 209) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x12cc5459-shared-pool-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x12cc5459-shared-pool-24 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-2034889633_15 at /127.0.0.1:53020 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:34053-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x12cc5459-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x12cc5459-shared-pool-23 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=551 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=377 (was 310) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=6852 (was 7526) 2023-04-18 18:54:09,654 INFO [Listener at localhost/36951] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=235, OpenFileDescriptor=551, MaxFileDescriptor=60000, SystemLoadAverage=377, ProcessCount=170, AvailableMemoryMB=6852 2023-04-18 18:54:09,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-18 18:54:09,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-18 18:54:09,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-18 18:54:09,659 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-18 18:54:09,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-18 18:54:09,660 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-18 18:54:09,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-18 18:54:09,662 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:09,662 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01 empty. 2023-04-18 18:54:09,663 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:09,663 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 18:54:09,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-18 18:54:09,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-18 18:54:10,073 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-18 18:54:10,075 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => cbf5abd278bb2be3a248e6824371aa01, NAME => 'TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp 2023-04-18 18:54:10,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-18 18:54:10,482 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 18:54:10,482 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing cbf5abd278bb2be3a248e6824371aa01, disabling compactions & flushes 2023-04-18 18:54:10,483 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:10,483 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:10,483 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. after waiting 0 ms 2023-04-18 18:54:10,483 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:10,483 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:10,483 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:10,485 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-18 18:54:10,486 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844050486"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681844050486"}]},"ts":"1681844050486"} 2023-04-18 18:54:10,488 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-18 18:54:10,488 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-18 18:54:10,489 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844050489"}]},"ts":"1681844050489"} 2023-04-18 18:54:10,490 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-18 18:54:10,504 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cbf5abd278bb2be3a248e6824371aa01, ASSIGN}] 2023-04-18 18:54:10,506 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cbf5abd278bb2be3a248e6824371aa01, ASSIGN 2023-04-18 18:54:10,507 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=cbf5abd278bb2be3a248e6824371aa01, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34053,1681844018952; forceNewPlan=false, retain=false 2023-04-18 18:54:10,658 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=cbf5abd278bb2be3a248e6824371aa01, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:10,658 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844050658"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681844050658"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681844050658"}]},"ts":"1681844050658"} 2023-04-18 18:54:10,659 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952}] 2023-04-18 18:54:10,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-18 18:54:10,815 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:10,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => cbf5abd278bb2be3a248e6824371aa01, NAME => 'TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.', STARTKEY => '', ENDKEY => ''} 2023-04-18 18:54:10,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:10,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 18:54:10,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:10,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:10,817 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:10,819 DEBUG [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A 2023-04-18 18:54:10,819 DEBUG [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A 2023-04-18 18:54:10,819 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:54:10,819 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cbf5abd278bb2be3a248e6824371aa01 columnFamilyName A 2023-04-18 18:54:10,820 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] regionserver.HStore(310): Store=cbf5abd278bb2be3a248e6824371aa01/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:54:10,820 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:10,821 DEBUG [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B 2023-04-18 18:54:10,821 DEBUG [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B 2023-04-18 18:54:10,822 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:54:10,822 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cbf5abd278bb2be3a248e6824371aa01 columnFamilyName B 2023-04-18 18:54:10,822 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] regionserver.HStore(310): Store=cbf5abd278bb2be3a248e6824371aa01/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:54:10,822 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:10,824 DEBUG [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C 2023-04-18 18:54:10,824 DEBUG [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C 2023-04-18 18:54:10,824 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:54:10,824 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cbf5abd278bb2be3a248e6824371aa01 columnFamilyName C 2023-04-18 18:54:10,824 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] regionserver.HStore(310): Store=cbf5abd278bb2be3a248e6824371aa01/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:54:10,825 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:10,825 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:10,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:10,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 18:54:10,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:10,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-18 18:54:10,831 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened cbf5abd278bb2be3a248e6824371aa01; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69036008, jitterRate=0.028716683387756348}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 18:54:10,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:10,832 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., pid=18, masterSystemTime=1681844050811 2023-04-18 18:54:10,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:10,833 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:10,834 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=cbf5abd278bb2be3a248e6824371aa01, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:10,834 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844050833"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681844050833"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681844050833"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681844050833"}]},"ts":"1681844050833"} 2023-04-18 18:54:10,836 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-18 18:54:10,836 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 in 176 msec 2023-04-18 18:54:10,838 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-18 18:54:10,838 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=cbf5abd278bb2be3a248e6824371aa01, ASSIGN in 332 msec 2023-04-18 18:54:10,839 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-18 18:54:10,839 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844050839"}]},"ts":"1681844050839"} 2023-04-18 18:54:10,840 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-18 18:54:10,843 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-18 18:54:10,844 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1860 sec 2023-04-18 18:54:11,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-18 18:54:11,766 INFO [Listener at localhost/36951] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-18 18:54:11,767 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x3ab850e7 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:54:11,773 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7053d0b0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:54:11,775 DEBUG [Listener at localhost/36951] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:54:11,779 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50430, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:54:11,786 DEBUG [Listener at localhost/36951] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-18 18:54:11,788 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33900, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-18 18:54:11,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-18 18:54:11,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-18 18:54:11,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-18 18:54:11,834 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-18 18:54:11,835 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-18 18:54:11,836 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-18 18:54:11,841 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-18 18:54:11,858 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cbf5abd278bb2be3a248e6824371aa01, REOPEN/MOVE}] 2023-04-18 18:54:11,859 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cbf5abd278bb2be3a248e6824371aa01, REOPEN/MOVE 2023-04-18 18:54:11,859 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=cbf5abd278bb2be3a248e6824371aa01, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:11,860 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844051859"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681844051859"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681844051859"}]},"ts":"1681844051859"} 2023-04-18 18:54:11,861 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952}] 2023-04-18 18:54:12,013 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:12,014 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing cbf5abd278bb2be3a248e6824371aa01, disabling compactions & flushes 2023-04-18 18:54:12,014 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:12,014 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:12,014 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. after waiting 0 ms 2023-04-18 18:54:12,014 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:12,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-18 18:54:12,019 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:12,019 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:12,019 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: cbf5abd278bb2be3a248e6824371aa01 to self. 2023-04-18 18:54:12,021 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:12,023 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=cbf5abd278bb2be3a248e6824371aa01, regionState=CLOSED 2023-04-18 18:54:12,023 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844052023"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681844052023"}]},"ts":"1681844052023"} 2023-04-18 18:54:12,026 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-18 18:54:12,026 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 in 163 msec 2023-04-18 18:54:12,027 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=cbf5abd278bb2be3a248e6824371aa01, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,34053,1681844018952; forceNewPlan=false, retain=true 2023-04-18 18:54:12,177 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=cbf5abd278bb2be3a248e6824371aa01, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,178 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844052177"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681844052177"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681844052177"}]},"ts":"1681844052177"} 2023-04-18 18:54:12,179 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952}] 2023-04-18 18:54:12,335 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:12,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => cbf5abd278bb2be3a248e6824371aa01, NAME => 'TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.', STARTKEY => '', ENDKEY => ''} 2023-04-18 18:54:12,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:12,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 18:54:12,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:12,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:12,340 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:12,341 DEBUG [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A 2023-04-18 18:54:12,341 DEBUG [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A 2023-04-18 18:54:12,341 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:54:12,345 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cbf5abd278bb2be3a248e6824371aa01 columnFamilyName A 2023-04-18 18:54:12,348 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] regionserver.HStore(310): Store=cbf5abd278bb2be3a248e6824371aa01/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:54:12,349 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:12,351 DEBUG [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B 2023-04-18 18:54:12,351 DEBUG [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B 2023-04-18 18:54:12,351 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:54:12,352 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cbf5abd278bb2be3a248e6824371aa01 columnFamilyName B 2023-04-18 18:54:12,355 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] regionserver.HStore(310): Store=cbf5abd278bb2be3a248e6824371aa01/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:54:12,355 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:12,356 DEBUG [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C 2023-04-18 18:54:12,356 DEBUG [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C 2023-04-18 18:54:12,356 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:54:12,357 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cbf5abd278bb2be3a248e6824371aa01 columnFamilyName C 2023-04-18 18:54:12,357 INFO [StoreOpener-cbf5abd278bb2be3a248e6824371aa01-1] regionserver.HStore(310): Store=cbf5abd278bb2be3a248e6824371aa01/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:54:12,358 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:12,358 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:12,360 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:12,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 18:54:12,364 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:12,365 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened cbf5abd278bb2be3a248e6824371aa01; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74191341, jitterRate=0.10553713142871857}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 18:54:12,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:12,366 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., pid=23, masterSystemTime=1681844052331 2023-04-18 18:54:12,367 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:12,367 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:12,368 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=cbf5abd278bb2be3a248e6824371aa01, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,368 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844052368"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681844052368"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681844052368"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681844052368"}]},"ts":"1681844052368"} 2023-04-18 18:54:12,371 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-18 18:54:12,371 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 in 191 msec 2023-04-18 18:54:12,373 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-18 18:54:12,373 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=cbf5abd278bb2be3a248e6824371aa01, REOPEN/MOVE in 513 msec 2023-04-18 18:54:12,375 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-18 18:54:12,375 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 535 msec 2023-04-18 18:54:12,378 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 575 msec 2023-04-18 18:54:12,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-18 18:54:12,388 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x63ebf1e9 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:54:12,393 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@695af552, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:54:12,394 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x52d98fed to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:54:12,398 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52f29016, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:54:12,398 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x18496e17 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:54:12,402 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f13c14, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:54:12,403 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x25441895 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:54:12,408 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a5a81b5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:54:12,408 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x40e0a3dc to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:54:12,412 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45e59be5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:54:12,412 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x2668d41d to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:54:12,416 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fbb4986, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:54:12,416 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x6b416c44 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:54:12,421 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b69850d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:54:12,422 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x204b5880 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:54:12,425 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40883381, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:54:12,426 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x41173aec to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:54:12,430 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3be2e516, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:54:12,436 DEBUG [hconnection-0x68f08b1b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:54:12,436 DEBUG [hconnection-0xaebc21-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:54:12,437 DEBUG [hconnection-0x78640d77-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:54:12,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:12,440 DEBUG [hconnection-0x46f6407-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:54:12,440 DEBUG [hconnection-0x215e00f9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:54:12,440 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50444, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:54:12,440 DEBUG [hconnection-0x1d11fbaf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:54:12,441 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50470, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:54:12,445 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50464, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:54:12,445 DEBUG [hconnection-0x54a2eca3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:54:12,445 DEBUG [hconnection-0x2a60706-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:54:12,448 DEBUG [hconnection-0x6faa509-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:54:12,448 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50448, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:54:12,449 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50512, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:54:12,449 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50484, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:54:12,452 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50514, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:54:12,454 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50496, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:54:12,455 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50520, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:54:12,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:12,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:12,459 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:12,459 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:12,459 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:12,459 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:12,459 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:12,459 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:12,461 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,461 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:12,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:12,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:12,461 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,461 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:12,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:12,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:12,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:12,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:12,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:12,471 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:12,471 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:12,471 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:12,472 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:12,472 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:12,474 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:12,474 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:12,474 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:12,474 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:12,481 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-18 18:54:12,484 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:12,484 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:12,485 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:12,485 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:12,485 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:12,485 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:12,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:12,492 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:12,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681844052500, max timestamp=1681844052517Num uniques 859; 2023-04-18 18:54:12,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681844052500, max timestamp=1681844052517Num uniques 859; 2023-04-18 18:54:12,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-18 18:54:12,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-18 18:54:12,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681844052500, max timestamp=1681844052517Num uniques 859; 2023-04-18 18:54:12,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-18 18:54:12,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:12,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50448 deadline: 1681844112541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:12,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50484 deadline: 1681844112541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:12,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50520 deadline: 1681844112542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:12,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844112544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,568 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418612bbe612d8c4b2092d7a38622dfcd7b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418612bbe612d8c4b2092d7a38622dfcd7b 2023-04-18 18:54:12,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:12,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50444 deadline: 1681844112555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,586 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=12, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/21e44d0ec751480780a1106dace9fade 2023-04-18 18:54:12,641 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/e78eda01e3834dc89c4c6696a487d30a 2023-04-18 18:54:12,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:12,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50484 deadline: 1681844112656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:12,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:12,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50520 deadline: 1681844112657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50448 deadline: 1681844112657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:12,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844112658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:12,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50444 deadline: 1681844112672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,687 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/708aeb4509414ca2b022536786c9dd80 2023-04-18 18:54:12,695 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/21e44d0ec751480780a1106dace9fade as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/21e44d0ec751480780a1106dace9fade 2023-04-18 18:54:12,702 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/21e44d0ec751480780a1106dace9fade, entries=100, sequenceid=12, filesize=18.4 K 2023-04-18 18:54:12,703 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/e78eda01e3834dc89c4c6696a487d30a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/e78eda01e3834dc89c4c6696a487d30a 2023-04-18 18:54:12,708 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/e78eda01e3834dc89c4c6696a487d30a, entries=100, sequenceid=12, filesize=9.3 K 2023-04-18 18:54:12,709 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/708aeb4509414ca2b022536786c9dd80 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/708aeb4509414ca2b022536786c9dd80 2023-04-18 18:54:12,716 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/708aeb4509414ca2b022536786c9dd80, entries=100, sequenceid=12, filesize=9.3 K 2023-04-18 18:54:12,718 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for cbf5abd278bb2be3a248e6824371aa01 in 237ms, sequenceid=12, compaction requested=false 2023-04-18 18:54:12,718 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-18 18:54:12,720 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:12,720 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:12,720 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:12,720 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:12,720 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,723 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:12,723 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:12,725 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:12,725 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:12,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:12,726 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:12,726 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:12,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:12,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:12,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:12,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:12,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:12,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66680865[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:12,729 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:12,729 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:12,731 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:12,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:12,731 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:12,731 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:12,731 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:12,731 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,732 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:12,733 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:12,733 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,733 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:12,733 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:12,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:12,736 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:12,736 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:12,736 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:12,737 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:12,737 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:12,738 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:12,738 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:12,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:12,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:12,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:12,738 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:12,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:12,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:12,739 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:12,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:12,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:12,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:12,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:12,744 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:12,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,747 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,747 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:12,747 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:12,747 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:12,747 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:12,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:12,747 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:12,747 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:12,748 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:12,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:12,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:12,748 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:12,748 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:12,748 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:12,748 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:12,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:12,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:12,848 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:12,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:12,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:12,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:12,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:12,856 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:12,856 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:12,856 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:12,856 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:12,857 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:12,857 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:12,859 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,859 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:12,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:12,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:12,859 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,859 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:12,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:12,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:12,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:12,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:12,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:12,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:12,864 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=161.02 KB heapSize=445.92 KB 2023-04-18 18:54:12,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:12,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:12,864 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:12,864 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:12,864 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:12,865 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:12,865 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:12,865 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:12,865 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:12,865 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:12,865 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:12,865 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:12,865 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:12,865 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,865 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:12,865 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:12,867 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:12,867 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:12,868 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:12,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681844052537, max timestamp=1681844052864Num uniques 1218; 2023-04-18 18:54:12,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681844052537, max timestamp=1681844052864Num uniques 1218; 2023-04-18 18:54:12,868 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:12,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=55778, new segment data size=9978, suffix heap size=154640,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1218, new segment cells count=218 2023-04-18 18:54:12,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:12,868 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:12,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681844052537, max timestamp=1681844052864Num uniques 1218; 2023-04-18 18:54:12,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=55778, new segment data size=9978, suffix heap size=154640,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1218, new segment cells count=218 2023-04-18 18:54:12,868 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:12,868 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:12,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=55778, new segment data size=9978, suffix heap size=154640,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1218, new segment cells count=218 2023-04-18 18:54:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:12,868 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:12,869 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:12,869 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:12,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:12,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@83408d[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:12,869 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:12,869 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:12,870 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:12,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:12,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:12,871 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:12,871 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:12,871 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:12,871 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,871 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:12,873 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:12,873 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,873 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:12,873 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:12,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:12,878 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:12,878 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:12,878 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:12,878 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:12,878 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:12,881 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:12,883 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:12,883 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:12,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:12,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:12,885 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:12,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:12,886 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:12,889 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:12,889 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,889 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:12,889 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:12,889 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:12,889 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,889 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:12,889 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:12,890 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:12,890 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:12,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:12,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:12,890 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:12,890 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:12,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:12,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:12,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:12,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:12,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844052887, max timestamp=1681844052891Num uniques 859; 2023-04-18 18:54:12,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:54:12,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844052887, max timestamp=1681844052891Num uniques 859; 2023-04-18 18:54:12,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:54:12,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844052887, max timestamp=1681844052891Num uniques 859; 2023-04-18 18:54:12,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:54:12,906 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304181de7af9c9b33451cbf2f041dc47bb10b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304181de7af9c9b33451cbf2f041dc47bb10b 2023-04-18 18:54:12,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:12,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844112918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:12,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50444 deadline: 1681844112919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:12,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7683, totalHeapSize=17712, min timestamp=1681844052916, max timestamp=1681844052919Num uniques 1068; 2023-04-18 18:54:12,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48903, new segment data size=7683, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-18 18:54:12,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7681, totalHeapSize=17712, min timestamp=1681844052916, max timestamp=1681844052919Num uniques 1068; 2023-04-18 18:54:12,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48901, new segment data size=7681, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-18 18:54:12,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7680, totalHeapSize=17712, min timestamp=1681844052916, max timestamp=1681844052919Num uniques 1068; 2023-04-18 18:54:12,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48900, new segment data size=7680, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-18 18:54:12,933 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/225aa18776e042c0b9bba0b5a5e15822 2023-04-18 18:54:12,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681844052957, max timestamp=1681844052962Num uniques 1027; 2023-04-18 18:54:12,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8092, totalHeapSize=18648, min timestamp=1681844052957, max timestamp=1681844052963Num uniques 1027; 2023-04-18 18:54:12,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 18:54:12,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47022, new segment data size=8092, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 18:54:12,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8099, totalHeapSize=18648, min timestamp=1681844052957, max timestamp=1681844052963Num uniques 1027; 2023-04-18 18:54:12,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47029, new segment data size=8099, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 18:54:12,990 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:12,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:12,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:13,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681844052987, max timestamp=1681844052995Num uniques 1036; 2023-04-18 18:54:13,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=6870, suffix heap size=135712,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=150 2023-04-18 18:54:13,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681844052987, max timestamp=1681844052995Num uniques 1036; 2023-04-18 18:54:13,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=6870, suffix heap size=135712,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=150 2023-04-18 18:54:13,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681844052987, max timestamp=1681844052995Num uniques 1036; 2023-04-18 18:54:13,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=6870, suffix heap size=135712,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=150 2023-04-18 18:54:13,007 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/161fd02fb99447dea3002ef9fbf41dee 2023-04-18 18:54:13,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=945, cellSize=43280, totalHeapSize=98520, min timestamp=1681844052987, max timestamp=1681844053036Num uniques 1009; 2023-04-18 18:54:13,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=43280, suffix heap size=132904,new segment heap size=98520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=945 2023-04-18 18:54:13,041 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=945, cellSize=43280, totalHeapSize=98520, min timestamp=1681844052987, max timestamp=1681844053036Num uniques 1009; 2023-04-18 18:54:13,041 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=43280, suffix heap size=132904,new segment heap size=98520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=945 2023-04-18 18:54:13,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681844053014, max timestamp=1681844053036Num uniques 1009; 2023-04-18 18:54:13,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=13510, suffix heap size=132904,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=295 2023-04-18 18:54:13,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:13,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:13,049 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:13,049 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:13,050 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:13,050 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:13,050 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,050 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:13,052 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:13,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:13,053 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:13,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:13,054 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:13,054 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,054 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:13,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50444 deadline: 1681844113059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50448 deadline: 1681844113059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:13,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:13,065 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:13,065 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:13,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,065 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:13,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50484 deadline: 1681844113062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50520 deadline: 1681844113060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844113063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,069 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:13,070 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:13,070 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:13,084 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:13,084 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:13,084 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:13,084 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:13,084 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:13,084 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,084 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:13,084 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:13,087 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,087 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:13,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:13,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:13,088 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,088 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:13,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:13,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:13,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:13,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:13,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79e160b2[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:13,090 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:13,090 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,092 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,092 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,092 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:13,092 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:13,092 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:13,092 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:13,094 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,094 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:13,094 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:13,094 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:13,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:13,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:13,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:13,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:13,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:13,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:13,097 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:13,097 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:13,097 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:13,097 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:13,097 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:13,105 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,105 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:13,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,105 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:13,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:13,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:13,106 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:13,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:13,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:13,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:13,108 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:13,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:13,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,111 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:13,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:13,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:13,111 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:13,111 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:13,112 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,112 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:13,112 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:13,112 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,112 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,112 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,112 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:13,112 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:13,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:13,112 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:13,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:13,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:13,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:13,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50444 deadline: 1681844113166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50448 deadline: 1681844113166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50484 deadline: 1681844113167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50520 deadline: 1681844113167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844113167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,212 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:13,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:13,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:13,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:13,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:13,220 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:13,220 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:13,220 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:13,220 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:13,220 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,220 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:13,222 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:13,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:13,222 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:13,222 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,223 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:13,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:13,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:13,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:13,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:13,224 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:13,224 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:13,224 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:13,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:13,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:13,232 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:13,232 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:13,232 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:13,232 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:13,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:13,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:13,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:13,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:13,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,237 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:13,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:13,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:13,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:13,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:13,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:13,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:13,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@497ad05a[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:13,241 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:13,241 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,243 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,243 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:13,243 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:13,243 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:13,243 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,244 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:13,246 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:13,246 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:13,246 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:13,246 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:13,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:13,251 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:13,251 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:13,251 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:13,251 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:13,251 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:13,253 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,253 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,254 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:13,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,255 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:13,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:13,258 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:13,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,262 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:13,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:13,262 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:13,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:13,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:13,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:13,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,263 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:13,263 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:13,263 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:13,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:13,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:13,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:13,363 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:13,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:13,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:13,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50444 deadline: 1681844113369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50448 deadline: 1681844113369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50520 deadline: 1681844113375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844113374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50484 deadline: 1681844113381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:13,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:13,416 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:13,416 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:13,416 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:13,416 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:13,417 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,417 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:13,419 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:13,419 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:13,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:13,419 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,420 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:13,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:13,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:13,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:13,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:13,432 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:13,432 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:13,432 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:13,436 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:13,436 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:13,436 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:13,436 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:13,436 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:13,436 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:13,436 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:13,436 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:13,436 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,436 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:13,436 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:13,438 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,438 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:13,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:13,438 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:13,438 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:13,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:13,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:13,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:13,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:13,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff7c230[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:13,440 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:13,440 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,441 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,442 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:13,442 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:13,442 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:13,442 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,442 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:13,444 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:13,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:13,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:13,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:13,448 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:13,448 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:13,448 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:13,448 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:13,448 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:13,450 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:13,450 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:13,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,452 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:13,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,456 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:13,456 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,456 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:13,456 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:13,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,456 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:13,457 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:13,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:13,457 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:13,457 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:13,457 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:13,457 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:13,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:13,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:13,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:13,457 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:13,457 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:13,457 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,457 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,458 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:13,458 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:13,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:13,485 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/d405f7c6c47848cfa990d6d4483503ab 2023-04-18 18:54:13,493 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/225aa18776e042c0b9bba0b5a5e15822 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/225aa18776e042c0b9bba0b5a5e15822 2023-04-18 18:54:13,500 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/225aa18776e042c0b9bba0b5a5e15822, entries=150, sequenceid=50, filesize=25.1 K 2023-04-18 18:54:13,502 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/161fd02fb99447dea3002ef9fbf41dee as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/161fd02fb99447dea3002ef9fbf41dee 2023-04-18 18:54:13,506 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/161fd02fb99447dea3002ef9fbf41dee, entries=150, sequenceid=50, filesize=11.6 K 2023-04-18 18:54:13,507 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/d405f7c6c47848cfa990d6d4483503ab as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/d405f7c6c47848cfa990d6d4483503ab 2023-04-18 18:54:13,514 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/d405f7c6c47848cfa990d6d4483503ab, entries=150, sequenceid=50, filesize=11.6 K 2023-04-18 18:54:13,515 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~80.58 KB/82512, currentSize=172.20 KB/176330 for cbf5abd278bb2be3a248e6824371aa01 in 651ms, sequenceid=50, compaction requested=false 2023-04-18 18:54:13,516 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:13,557 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:13,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:13,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:13,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:13,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:13,568 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:13,568 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:13,568 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:13,568 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:13,569 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,569 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:13,571 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:13,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:13,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:13,572 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:13,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:13,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:13,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:13,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:13,574 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,574 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:13,574 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:13,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:13,574 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:13,574 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:13,574 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:13,576 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:13,576 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:13,576 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:13,576 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:13,576 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=172.20 KB heapSize=444.50 KB 2023-04-18 18:54:13,576 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:13,576 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:13,576 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:13,577 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:13,577 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:13,577 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:13,651 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304180a6b8e0446f2488f8c7bd17f6da740d3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304180a6b8e0446f2488f8c7bd17f6da740d3 2023-04-18 18:54:13,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:13,675 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:13,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50520 deadline: 1681844113681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844113682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50444 deadline: 1681844113682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50448 deadline: 1681844113682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50484 deadline: 1681844113699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50444 deadline: 1681844113790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50448 deadline: 1681844113790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50448 deadline: 1681844113994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:13,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:13,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50444 deadline: 1681844113996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,080 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=150, memsize=67.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/99837c2234fd48f49150539f64d97907 2023-04-18 18:54:14,115 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=67.09 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/225ef43b3d884550ac9c6c9ade3e0a2e 2023-04-18 18:54:14,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:14,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50520 deadline: 1681844114185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:14,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844114192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:14,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50484 deadline: 1681844114203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:14,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50448 deadline: 1681844114296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:14,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50444 deadline: 1681844114302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,371 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-18 18:54:14,537 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/219231f16fa04e51950b75f1bb5def8b 2023-04-18 18:54:14,550 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/99837c2234fd48f49150539f64d97907 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/99837c2234fd48f49150539f64d97907 2023-04-18 18:54:14,557 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/99837c2234fd48f49150539f64d97907, entries=150, sequenceid=150, filesize=25.3 K 2023-04-18 18:54:14,559 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/225ef43b3d884550ac9c6c9ade3e0a2e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/225ef43b3d884550ac9c6c9ade3e0a2e 2023-04-18 18:54:14,565 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/225ef43b3d884550ac9c6c9ade3e0a2e, entries=150, sequenceid=150, filesize=11.8 K 2023-04-18 18:54:14,566 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/219231f16fa04e51950b75f1bb5def8b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/219231f16fa04e51950b75f1bb5def8b 2023-04-18 18:54:14,573 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/219231f16fa04e51950b75f1bb5def8b, entries=150, sequenceid=150, filesize=11.8 K 2023-04-18 18:54:14,574 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~172.20 KB/176330, heapSize ~444.45 KB/455120, currentSize=26.84 KB/27480 for cbf5abd278bb2be3a248e6824371aa01 in 998ms, sequenceid=150, compaction requested=true 2023-04-18 18:54:14,574 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:14,574 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:14,575 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:14,575 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:14,575 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,575 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:14,575 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:14,579 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:14,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:14,581 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:14,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:14,583 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:14,583 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:14,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:14,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:14,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75ea606b[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:14,585 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:14,585 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:14,586 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:14,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:14,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:14,586 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:14,586 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:14,586 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:14,587 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,587 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:14,589 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:14,589 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,589 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:14,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:14,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:14,590 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:14,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:14,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:14,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:14,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:14,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:14,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:14,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:14,594 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:14,594 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:14,594 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:14,594 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:14,594 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:14,598 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:14,598 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:14,598 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:14,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:14,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:14,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:14,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:14,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:14,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:14,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:14,599 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:14,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:14,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:14,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:14,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:14,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,600 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:14,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:14,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,607 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:14,607 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:14,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:14,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:14,608 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,608 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:14,608 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:14,608 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:14,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:14,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:14,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:14,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:14,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:14,608 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,608 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:14,608 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:14,609 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:14,609 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:14,609 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:14,709 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:14,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:14,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:14,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:14,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:14,712 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:14,713 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:14,713 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:14,713 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:14,713 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:14,713 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:14,714 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:14,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:14,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:14,714 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:14,715 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:14,715 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:14,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:14,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:14,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:14,716 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:14,716 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:14,716 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:14,716 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:14,716 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:14,716 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:14,717 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:14,717 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:14,717 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:14,717 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:14,717 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-18 18:54:14,717 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:14,717 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:14,717 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:14,717 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:14,717 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:14,717 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:14,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:14,813 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:14,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844054831, max timestamp=1681844054840Num uniques 859; 2023-04-18 18:54:14,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844054831, max timestamp=1681844054840Num uniques 859; 2023-04-18 18:54:14,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:54:14,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:54:14,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844054831, max timestamp=1681844054840Num uniques 859; 2023-04-18 18:54:14,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:54:14,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681844054862, max timestamp=1681844054887Num uniques 1068; 2023-04-18 18:54:14,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-18 18:54:14,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681844054862, max timestamp=1681844054887Num uniques 1068; 2023-04-18 18:54:14,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-18 18:54:14,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681844054862, max timestamp=1681844054887Num uniques 1068; 2023-04-18 18:54:14,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-18 18:54:14,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681844054899, max timestamp=1681844054935Num uniques 1127; 2023-04-18 18:54:14,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-04-18 18:54:14,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681844054899, max timestamp=1681844054935Num uniques 1127; 2023-04-18 18:54:14,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-04-18 18:54:14,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681844054899, max timestamp=1681844054935Num uniques 1127; 2023-04-18 18:54:14,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-04-18 18:54:14,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681844054977, max timestamp=1681844054990Num uniques 1136; 2023-04-18 18:54:14,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-04-18 18:54:14,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681844054977, max timestamp=1681844054990Num uniques 1136; 2023-04-18 18:54:14,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-04-18 18:54:14,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681844054977, max timestamp=1681844054990Num uniques 1136; 2023-04-18 18:54:14,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-04-18 18:54:15,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681844055014, max timestamp=1681844055030Num uniques 1145; 2023-04-18 18:54:15,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-18 18:54:15,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681844055014, max timestamp=1681844055030Num uniques 1145; 2023-04-18 18:54:15,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-18 18:54:15,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681844055014, max timestamp=1681844055030Num uniques 1145; 2023-04-18 18:54:15,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-18 18:54:15,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681844055063, max timestamp=1681844055076Num uniques 1104; 2023-04-18 18:54:15,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-18 18:54:15,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681844055063, max timestamp=1681844055079Num uniques 1104; 2023-04-18 18:54:15,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681844055063, max timestamp=1681844055076Num uniques 1104; 2023-04-18 18:54:15,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-18 18:54:15,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-18 18:54:15,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681844055098, max timestamp=1681844055129Num uniques 1063; 2023-04-18 18:54:15,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-18 18:54:15,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681844055098, max timestamp=1681844055129Num uniques 1063; 2023-04-18 18:54:15,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-18 18:54:15,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681844055098, max timestamp=1681844055129Num uniques 1063; 2023-04-18 18:54:15,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-18 18:54:15,135 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304181a58efce06a74f96bfa9e3ff986ef621 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304181a58efce06a74f96bfa9e3ff986ef621 2023-04-18 18:54:15,147 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=157, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/68a6066a143a47d5ad7ee6026a21309a 2023-04-18 18:54:15,169 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/aea431548e3646b69982c165eb2532be 2023-04-18 18:54:15,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1681844055179, max timestamp=1681844055195Num uniques 1122; 2023-04-18 18:54:15,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=14742, suffix heap size=144656,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=322 2023-04-18 18:54:15,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1681844055179, max timestamp=1681844055195Num uniques 1122; 2023-04-18 18:54:15,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=14742, suffix heap size=144656,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=322 2023-04-18 18:54:15,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681844055180, max timestamp=1681844055195Num uniques 1122; 2023-04-18 18:54:15,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-04-18 18:54:15,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:15,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844115221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681844055219, max timestamp=1681844055230Num uniques 1181; 2023-04-18 18:54:15,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=8286, suffix heap size=150792,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=181 2023-04-18 18:54:15,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681844055219, max timestamp=1681844055230Num uniques 1031; 2023-04-18 18:54:15,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-18 18:54:15,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681844055219, max timestamp=1681844055230Num uniques 1181; 2023-04-18 18:54:15,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=8286, suffix heap size=150792,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=181 2023-04-18 18:54:15,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681844055276, max timestamp=1681844055283Num uniques 1040; 2023-04-18 18:54:15,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-18 18:54:15,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681844055276, max timestamp=1681844055283Num uniques 1040; 2023-04-18 18:54:15,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681844055276, max timestamp=1681844055283Num uniques 1040; 2023-04-18 18:54:15,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-18 18:54:15,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-18 18:54:15,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:15,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844115334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:15,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681844055311, max timestamp=1681844055335Num uniques 1099; 2023-04-18 18:54:15,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50520 deadline: 1681844115335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-18 18:54:15,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681844055311, max timestamp=1681844055335Num uniques 1099; 2023-04-18 18:54:15,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-18 18:54:15,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9104, totalHeapSize=20936, min timestamp=1681844055311, max timestamp=1681844055336Num uniques 1099; 2023-04-18 18:54:15,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50324, new segment data size=9104, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-18 18:54:15,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681844055353, max timestamp=1681844055391Num uniques 1058; 2023-04-18 18:54:15,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681844055353, max timestamp=1681844055391Num uniques 1058; 2023-04-18 18:54:15,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681844055353, max timestamp=1681844055391Num uniques 1058; 2023-04-18 18:54:15,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-04-18 18:54:15,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-04-18 18:54:15,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-04-18 18:54:15,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681844055413, max timestamp=1681844055422Num uniques 1117; 2023-04-18 18:54:15,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-18 18:54:15,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681844055413, max timestamp=1681844055422Num uniques 1117; 2023-04-18 18:54:15,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-18 18:54:15,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681844055413, max timestamp=1681844055422Num uniques 1117; 2023-04-18 18:54:15,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-18 18:54:15,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1681844055440, max timestamp=1681844055450Num uniques 1076; 2023-04-18 18:54:15,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=12636, suffix heap size=139872,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=276 2023-04-18 18:54:15,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1681844055440, max timestamp=1681844055450Num uniques 1076; 2023-04-18 18:54:15,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=12636, suffix heap size=139872,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=276 2023-04-18 18:54:15,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1681844055440, max timestamp=1681844055450Num uniques 1076; 2023-04-18 18:54:15,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=12636, suffix heap size=139872,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=276 2023-04-18 18:54:15,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:15,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50520 deadline: 1681844115479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:15,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844115480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681844055470, max timestamp=1681844055479Num uniques 1135; 2023-04-18 18:54:15,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681844055470, max timestamp=1681844055479Num uniques 1135; 2023-04-18 18:54:15,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=8470, suffix heap size=146008,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=185 2023-04-18 18:54:15,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=8470, suffix heap size=146008,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=185 2023-04-18 18:54:15,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681844055470, max timestamp=1681844055479Num uniques 1135; 2023-04-18 18:54:15,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=8470, suffix heap size=146008,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=185 2023-04-18 18:54:15,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1681844055498, max timestamp=1681844055526Num uniques 1044; 2023-04-18 18:54:15,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1681844055498, max timestamp=1681844055526Num uniques 1044; 2023-04-18 18:54:15,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1681844055498, max timestamp=1681844055526Num uniques 1044; 2023-04-18 18:54:15,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=13464, suffix heap size=136544,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=294 2023-04-18 18:54:15,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=13464, suffix heap size=136544,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=294 2023-04-18 18:54:15,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=13464, suffix heap size=136544,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=294 2023-04-18 18:54:15,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:15,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844115556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:15,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50484 deadline: 1681844115558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:15,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844115558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:15,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844115583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:15,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50520 deadline: 1681844115584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,609 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/dc8d1b54b39840aa958407bdf20342b3 2023-04-18 18:54:15,623 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/68a6066a143a47d5ad7ee6026a21309a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/68a6066a143a47d5ad7ee6026a21309a 2023-04-18 18:54:15,633 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/68a6066a143a47d5ad7ee6026a21309a, entries=150, sequenceid=157, filesize=25.3 K 2023-04-18 18:54:15,634 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/aea431548e3646b69982c165eb2532be as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/aea431548e3646b69982c165eb2532be 2023-04-18 18:54:15,642 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/aea431548e3646b69982c165eb2532be, entries=150, sequenceid=157, filesize=11.8 K 2023-04-18 18:54:15,643 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/dc8d1b54b39840aa958407bdf20342b3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/dc8d1b54b39840aa958407bdf20342b3 2023-04-18 18:54:15,652 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/dc8d1b54b39840aa958407bdf20342b3, entries=150, sequenceid=157, filesize=11.8 K 2023-04-18 18:54:15,654 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=154.31 KB/158010 for cbf5abd278bb2be3a248e6824371aa01 in 936ms, sequenceid=157, compaction requested=true 2023-04-18 18:54:15,654 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:15,654 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:15,654 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:15,654 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:15,654 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,654 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:15,654 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:15,657 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,657 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:15,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:15,658 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,658 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:15,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:15,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:15,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:15,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:15,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:15,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fc6a94[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:15,660 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:15,660 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,661 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,661 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:15,661 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:15,661 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:15,662 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,662 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:15,663 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:15,663 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:15,663 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:15,663 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:15,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:15,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:15,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:15,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:15,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:15,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:15,666 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=161.02 KB heapSize=452.72 KB 2023-04-18 18:54:15,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:15,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681844055555, max timestamp=1681844055665Num uniques 1153; 2023-04-18 18:54:15,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:15,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:15,667 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:15,668 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:15,668 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 18:54:15,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=7005, suffix heap size=147880,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=153 2023-04-18 18:54:15,668 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:15,668 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:15,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681844055555, max timestamp=1681844055665Num uniques 1153; 2023-04-18 18:54:15,668 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:15,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=7005, suffix heap size=147880,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=153 2023-04-18 18:54:15,668 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:15,668 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:15,668 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:15,668 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:15,668 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:15,671 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,671 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:15,671 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:15,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:15,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:15,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:15,672 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:15,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:15,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:15,673 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:15,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:15,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:15,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,677 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:15,677 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:15,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:15,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:15,677 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,677 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:15,677 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:15,677 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,678 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,678 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,678 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,678 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:15,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:15,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:15,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:15,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:15,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:15,678 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:15,678 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:15,707 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418a19570c1167e48709d6ae45db43df48e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a19570c1167e48709d6ae45db43df48e 2023-04-18 18:54:15,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:15,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844115717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:15,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50484 deadline: 1681844115733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:15,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844115733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,778 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:15,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:15,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:15,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:15,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:15,785 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:15,785 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:15,785 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:15,785 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:15,785 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,785 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:15,787 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:15,787 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:15,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:15,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:15,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844115787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,789 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,789 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:15,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:15,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:15,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:15,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:15,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:15,790 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:15,790 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:15,790 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:15,790 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:15,790 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:15,790 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:15,790 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:15,790 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:15,790 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:15,790 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:15,790 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:15,790 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,791 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:15,791 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:15,792 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:15,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:15,792 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:15,793 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,793 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:15,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:15,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:15,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:15,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:15,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@181b9231[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:15,794 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:15,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:15,794 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50520 deadline: 1681844115789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,797 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,797 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:15,797 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:15,797 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:15,797 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,797 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:15,800 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:15,800 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:15,800 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:15,800 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:15,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:15,803 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:15,803 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:15,803 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:15,803 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:15,803 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:15,806 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,806 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:15,806 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:15,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:15,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:15,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:15,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:15,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:15,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:15,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,811 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,812 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,815 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,816 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:15,816 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:15,816 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:15,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:15,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:15,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:15,816 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:15,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:15,816 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:15,816 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,816 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,816 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,816 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,816 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:15,816 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:15,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:15,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:15,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:15,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844115836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:15,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844115839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:15,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50484 deadline: 1681844115839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,916 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:15,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:15,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:15,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:15,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:15,927 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:15,927 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:15,927 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:15,927 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:15,927 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,928 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:15,929 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:15,929 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:15,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:15,929 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,929 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:15,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:15,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:15,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:15,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:15,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:15,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:15,931 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:15,931 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:15,931 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:15,931 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:15,931 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:15,931 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:15,931 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:15,931 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:15,931 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:15,931 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:15,932 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:15,932 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,932 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:15,932 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:15,933 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,933 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:15,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:15,934 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:15,934 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:15,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:15,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:15,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@492c3540[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:15,936 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:15,936 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,938 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,938 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:15,938 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:15,938 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:15,939 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,939 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:15,940 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,940 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:15,940 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:15,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:15,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:15,940 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:15,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:15,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:15,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:15,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:15,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:15,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:15,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:15,943 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:15,943 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:15,943 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:15,943 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:15,943 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:15,945 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,945 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:15,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:15,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,945 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:15,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:15,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:15,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:15,946 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:15,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:15,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:15,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,948 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:15,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:15,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,950 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,950 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:15,950 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:15,950 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:15,950 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,950 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:15,950 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:15,950 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:15,950 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:15,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:15,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:15,950 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:15,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:15,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:15,951 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:15,951 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:15,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:15,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:16,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:16,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844116041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:16,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844116041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:16,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50484 deadline: 1681844116045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,051 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:16,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:16,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:16,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:16,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:16,056 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:16,056 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:16,056 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:16,056 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:16,057 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,057 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,064 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,064 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:16,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:16,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:16,065 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,065 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:16,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:16,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:16,066 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:16,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:16,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:16,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:16,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:16,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:16,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:16,066 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:16,066 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:16,066 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:16,066 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:16,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:16,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:16,067 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,067 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:16,067 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:16,068 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:16,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:16,068 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:16,069 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,069 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:16,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:16,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:16,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:16,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f63fd36[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:16,070 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:16,070 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,071 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,072 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:16,072 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:16,072 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:16,072 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,072 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:16,073 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:16,073 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:16,073 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,073 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:16,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:16,075 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:16,076 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:16,076 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:16,076 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:16,076 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:16,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:16,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:16,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:16,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:16,078 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,080 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,082 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,082 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,082 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,082 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,082 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,082 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,082 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,082 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:16,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:16,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:16,082 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:16,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:16,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:16,082 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:16,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:16,083 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:16,083 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:16,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:16,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:16,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:16,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844116089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:16,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50520 deadline: 1681844116096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,115 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=454, memsize=55.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/4378f2e643d94e8188ede4a98c3796eb 2023-04-18 18:54:16,140 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/27365794d38f469186d7ee8e373f0bc4 2023-04-18 18:54:16,155 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/b0ebce8ccfa748f5acda15bc88ab73f2 2023-04-18 18:54:16,162 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/4378f2e643d94e8188ede4a98c3796eb as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/4378f2e643d94e8188ede4a98c3796eb 2023-04-18 18:54:16,174 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/4378f2e643d94e8188ede4a98c3796eb, entries=250, sequenceid=454, filesize=39.2 K 2023-04-18 18:54:16,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/27365794d38f469186d7ee8e373f0bc4 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/27365794d38f469186d7ee8e373f0bc4 2023-04-18 18:54:16,183 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:16,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:16,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:16,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:16,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:16,189 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:16,189 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:16,189 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:16,189 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:16,193 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/27365794d38f469186d7ee8e373f0bc4, entries=150, sequenceid=454, filesize=11.9 K 2023-04-18 18:54:16,194 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,194 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,195 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/b0ebce8ccfa748f5acda15bc88ab73f2 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/b0ebce8ccfa748f5acda15bc88ab73f2 2023-04-18 18:54:16,196 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:16,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:16,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:16,197 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,197 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,197 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:16,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:16,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:16,205 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:16,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:16,206 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:16,206 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:16,206 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:16,218 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:16,219 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:16,219 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:16,219 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:16,219 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:16,219 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:16,219 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:16,219 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:16,219 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:16,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:16,234 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,234 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:16,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:16,235 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,235 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:16,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:16,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,236 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/b0ebce8ccfa748f5acda15bc88ab73f2, entries=150, sequenceid=454, filesize=11.9 K 2023-04-18 18:54:16,237 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~78.27 KB/80150, heapSize ~215.31 KB/220480, currentSize=107.34 KB/109920 for cbf5abd278bb2be3a248e6824371aa01 in 571ms, sequenceid=454, compaction requested=true 2023-04-18 18:54:16,237 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:16,238 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:16,238 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:54:16,238 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:54:16,238 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:16,238 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:16,241 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57686 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:54:16,241 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): cbf5abd278bb2be3a248e6824371aa01/B is initiating minor compaction (all files) 2023-04-18 18:54:16,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:16,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b2f9284[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:16,241 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 136466 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:54:16,241 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cbf5abd278bb2be3a248e6824371aa01/B in TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:16,241 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): cbf5abd278bb2be3a248e6824371aa01/A is initiating minor compaction (all files) 2023-04-18 18:54:16,241 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cbf5abd278bb2be3a248e6824371aa01/A in TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:16,241 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/e78eda01e3834dc89c4c6696a487d30a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/161fd02fb99447dea3002ef9fbf41dee, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/225ef43b3d884550ac9c6c9ade3e0a2e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/aea431548e3646b69982c165eb2532be, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/27365794d38f469186d7ee8e373f0bc4] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp, totalSize=56.3 K 2023-04-18 18:54:16,241 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/21e44d0ec751480780a1106dace9fade, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/225aa18776e042c0b9bba0b5a5e15822, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/99837c2234fd48f49150539f64d97907, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/68a6066a143a47d5ad7ee6026a21309a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/4378f2e643d94e8188ede4a98c3796eb] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp, totalSize=133.3 K 2023-04-18 18:54:16,241 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:16,241 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,242 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting e78eda01e3834dc89c4c6696a487d30a, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681844052474 2023-04-18 18:54:16,243 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 161fd02fb99447dea3002ef9fbf41dee, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681844052545 2023-04-18 18:54:16,243 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 225ef43b3d884550ac9c6c9ade3e0a2e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681844053048 2023-04-18 18:54:16,243 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting aea431548e3646b69982c165eb2532be, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681844053674 2023-04-18 18:54:16,244 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 21e44d0ec751480780a1106dace9fade, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681844052474 2023-04-18 18:54:16,244 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 27365794d38f469186d7ee8e373f0bc4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681844055556 2023-04-18 18:54:16,248 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,248 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:16,248 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:16,248 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:16,248 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 225aa18776e042c0b9bba0b5a5e15822, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681844052545 2023-04-18 18:54:16,249 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,249 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:16,249 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 99837c2234fd48f49150539f64d97907, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681844053048 2023-04-18 18:54:16,249 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 68a6066a143a47d5ad7ee6026a21309a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681844053674 2023-04-18 18:54:16,250 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 4378f2e643d94e8188ede4a98c3796eb, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681844055555 2023-04-18 18:54:16,250 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,250 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:16,250 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:16,250 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:16,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:16,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:16,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:16,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:16,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:16,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:16,254 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:16,254 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:16,255 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:16,255 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:16,255 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:16,257 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,257 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,257 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:16,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:16,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:16,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:16,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:16,258 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:16,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:16,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,259 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:16,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:16,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:16,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,264 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:16,264 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:16,263 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:16,264 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,264 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:16,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:16,264 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:16,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:16,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:16,282 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): cbf5abd278bb2be3a248e6824371aa01#B#compaction#129 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:16,305 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): cbf5abd278bb2be3a248e6824371aa01#A#compaction#128 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:16,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:16,345 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 18:54:16,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:16,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:16,346 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:16,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:16,346 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:16,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:16,364 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:16,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:16,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:16,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:16,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844116368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:16,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50484 deadline: 1681844116377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:16,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844116381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:16,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:16,393 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:16,393 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:16,393 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:16,393 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:16,394 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,394 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,397 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,397 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:16,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:16,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:16,398 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,398 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:16,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:16,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:16,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:16,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:16,413 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:16,413 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:16,413 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:16,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:16,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:16,415 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:16,415 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:16,415 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:16,415 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:16,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:16,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:16,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:16,422 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,422 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:16,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:16,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:16,423 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,423 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:16,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:16,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f6113ca[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:16,425 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:16,425 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,427 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,427 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:16,427 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:16,427 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:16,428 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,428 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:16,429 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304181ea46204c5bd48a28bce372228bd1a21 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304181ea46204c5bd48a28bce372228bd1a21 2023-04-18 18:54:16,430 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:16,430 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:16,430 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:16,430 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:16,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:16,435 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:16,435 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:16,435 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:16,436 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:16,436 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:16,440 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,440 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:16,440 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:16,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:16,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:16,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:16,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:16,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,441 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:16,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,442 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:16,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,445 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:16,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:16,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:16,445 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:16,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:16,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:16,452 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,452 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,452 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,452 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,452 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,452 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,452 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,452 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:16,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:16,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:16,453 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:16,453 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:16,464 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=474, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/79acfe2c6e854ea2a43f219bb6faf9f0 2023-04-18 18:54:16,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:16,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50484 deadline: 1681844116482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:16,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844116482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:16,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844116483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,553 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:16,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:16,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:16,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:16,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:16,559 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:16,559 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:16,559 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:16,559 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:16,560 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,560 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,562 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,562 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:16,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:16,563 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,563 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:16,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:16,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:16,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:16,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:16,564 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:16,565 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:16,565 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:16,565 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:16,565 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:16,565 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:16,565 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:16,565 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:16,565 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:16,565 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:16,565 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:16,565 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,565 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:16,565 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:16,570 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:16,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:16,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:16,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:16,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,572 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,572 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,572 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:16,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:16,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:16,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@674fa953[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:16,573 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:16,573 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,574 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,575 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:16,575 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:16,575 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:16,575 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,575 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:16,576 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:16,577 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:16,577 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:16,577 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:16,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:16,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:16,579 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:16,579 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:16,579 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:16,579 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:16,579 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:16,582 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,582 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:16,582 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,583 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:16,585 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:16,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:16,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844116592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:16,596 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:16,596 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:16,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:16,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:16,597 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,597 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:16,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:16,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:16,597 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,597 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:16,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:16,597 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,598 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,598 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,598 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,598 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:16,598 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:16,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:16,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:16,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50520 deadline: 1681844116600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:16,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50484 deadline: 1681844116685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:16,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844116685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:16,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844116686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,698 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:16,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:16,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:16,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:16,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:16,716 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:16,716 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:16,716 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:16,716 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:16,717 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,717 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,718 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:16,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:16,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:16,718 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,719 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,719 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:16,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:16,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:16,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:16,724 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:16,724 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:16,724 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:16,734 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:16,734 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:16,734 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:16,734 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:16,734 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:16,734 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:16,734 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:16,734 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:16,734 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,734 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:16,734 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:16,736 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,738 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:16,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:16,738 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,738 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:16,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:16,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:16,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17b572c1[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:16,740 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:16,741 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,742 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,742 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:16,742 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:16,742 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:16,744 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,744 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:16,745 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:16,745 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:16,745 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:16,746 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:16,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:16,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:16,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,752 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/176501c0f2d040fc8f0ee73d8588f1b3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/176501c0f2d040fc8f0ee73d8588f1b3 2023-04-18 18:54:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:16,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:16,755 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:16,755 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:16,755 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:16,755 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:16,755 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:16,761 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,761 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:16,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,761 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:16,762 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,763 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in cbf5abd278bb2be3a248e6824371aa01/B of cbf5abd278bb2be3a248e6824371aa01 into 176501c0f2d040fc8f0ee73d8588f1b3(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:16,763 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:16,763 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., storeName=cbf5abd278bb2be3a248e6824371aa01/B, priority=11, startTime=1681844056238; duration=0sec 2023-04-18 18:54:16,763 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:16,763 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:54:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:16,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,765 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57686 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:54:16,765 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): cbf5abd278bb2be3a248e6824371aa01/C is initiating minor compaction (all files) 2023-04-18 18:54:16,766 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cbf5abd278bb2be3a248e6824371aa01/C in TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:16,766 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/708aeb4509414ca2b022536786c9dd80, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/d405f7c6c47848cfa990d6d4483503ab, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/219231f16fa04e51950b75f1bb5def8b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/dc8d1b54b39840aa958407bdf20342b3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/b0ebce8ccfa748f5acda15bc88ab73f2] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp, totalSize=56.3 K 2023-04-18 18:54:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:16,766 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 708aeb4509414ca2b022536786c9dd80, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681844052474 2023-04-18 18:54:16,767 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,767 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting d405f7c6c47848cfa990d6d4483503ab, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681844052545 2023-04-18 18:54:16,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,769 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 219231f16fa04e51950b75f1bb5def8b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681844053048 2023-04-18 18:54:16,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:16,770 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting dc8d1b54b39840aa958407bdf20342b3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681844053674 2023-04-18 18:54:16,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,770 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b0ebce8ccfa748f5acda15bc88ab73f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681844055556 2023-04-18 18:54:16,771 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,771 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,771 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,771 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,771 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,771 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,771 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,771 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:16,771 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:16,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:16,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:16,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:16,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:16,771 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:16,772 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:16,772 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:16,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:16,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:16,787 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): cbf5abd278bb2be3a248e6824371aa01#C#compaction#132 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:16,850 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/97234ea094904e52aba1596f22b2f652 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/97234ea094904e52aba1596f22b2f652 2023-04-18 18:54:16,858 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in cbf5abd278bb2be3a248e6824371aa01/A of cbf5abd278bb2be3a248e6824371aa01 into 97234ea094904e52aba1596f22b2f652(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:16,858 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:16,858 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., storeName=cbf5abd278bb2be3a248e6824371aa01/A, priority=11, startTime=1681844056238; duration=0sec 2023-04-18 18:54:16,858 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:16,872 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:16,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:16,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:16,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:16,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:16,877 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:16,877 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:16,877 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:16,877 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:16,878 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,878 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,879 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:16,880 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,880 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,880 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:16,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:16,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:16,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:16,881 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:16,881 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:16,882 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:16,884 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:16,884 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:16,884 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:16,884 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:16,884 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:16,884 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:16,884 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:16,884 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:16,884 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,884 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:16,884 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:16,887 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,887 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:16,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:16,887 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,887 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:16,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:16,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:16,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:16,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:16,890 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:16,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60f96901[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:16,890 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,891 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,891 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:16,893 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:16,893 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:16,894 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,894 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:16,895 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:16,895 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:16,895 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:16,895 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:16,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:16,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:16,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:16,898 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/894b8e318d214f18b9bf0ed96bc87077 2023-04-18 18:54:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:16,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:16,900 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:16,900 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:16,900 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:16,900 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:16,900 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:16,907 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,908 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,911 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,911 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,911 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,911 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:16,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:16,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:16,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,911 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:16,911 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,911 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:16,911 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,911 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:16,911 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:16,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:16,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:16,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:16,911 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:16,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:16,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:16,911 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,912 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:16,912 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:16,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:16,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:16,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:16,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:16,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:16,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:16,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:16,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50484 deadline: 1681844116996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:16,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:16,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844116996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:17,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844116999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,012 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:17,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:17,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:17,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:17,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:17,020 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:17,020 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:17,020 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:17,020 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:17,021 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,021 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:17,022 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:17,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:17,022 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:17,023 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,023 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:17,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:17,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:17,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:17,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:17,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:17,028 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:17,028 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:17,028 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:17,028 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:17,028 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:17,029 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:17,029 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:17,029 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:17,029 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:17,029 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:17,029 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:17,029 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,029 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:17,029 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:17,031 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,031 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:17,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:17,031 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,031 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:17,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:17,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:17,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:17,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:17,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:17,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2741d997[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:17,033 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:17,033 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,034 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,034 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:17,034 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:17,034 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:17,035 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,035 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:17,044 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,044 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:17,044 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:17,044 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:17,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:17,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:17,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:17,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:17,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:17,046 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:17,046 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:17,046 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:17,046 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:17,046 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:17,048 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,048 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:17,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,048 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:17,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:17,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:17,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:17,048 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:17,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:17,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:17,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:17,050 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:17,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:17,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,052 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,052 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:17,052 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:17,052 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:17,052 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:17,052 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:17,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:17,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:17,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:17,052 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:17,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:17,053 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,053 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:17,053 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:17,053 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,053 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:17,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:17,153 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:17,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:17,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:17,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:17,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:17,157 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:17,157 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:17,157 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:17,157 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:17,157 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,157 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:17,159 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:17,159 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:17,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:17,159 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,159 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:17,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:17,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:17,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:17,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:17,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:17,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:17,160 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:17,160 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:17,160 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:17,164 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:17,164 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:17,164 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:17,164 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:17,164 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:17,164 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:17,164 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:17,164 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:17,164 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,164 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:17,164 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:17,171 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,171 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:17,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:17,172 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,172 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:17,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:17,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:17,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:17,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:17,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:17,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ddee91d[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:17,177 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:17,177 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,179 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,179 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:17,179 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:17,179 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:17,179 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,179 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:17,181 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,181 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:17,181 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:17,181 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:17,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:17,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:17,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:17,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:17,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:17,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:17,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:17,191 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:17,191 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:17,191 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:17,191 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:17,191 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:17,193 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:17,194 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:17,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:17,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,197 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,199 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,199 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:17,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,202 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,202 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:17,203 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:17,203 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:17,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:17,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:17,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:17,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:17,203 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:17,203 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,203 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:17,203 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:17,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:17,203 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:17,203 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,204 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,217 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/20f130deb76344ad9f0bf0e979848848 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/20f130deb76344ad9f0bf0e979848848 2023-04-18 18:54:17,222 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in cbf5abd278bb2be3a248e6824371aa01/C of cbf5abd278bb2be3a248e6824371aa01 into 20f130deb76344ad9f0bf0e979848848(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:17,222 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:17,222 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., storeName=cbf5abd278bb2be3a248e6824371aa01/C, priority=11, startTime=1681844056238; duration=0sec 2023-04-18 18:54:17,222 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:17,303 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:17,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:17,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:17,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:17,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:17,307 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:17,307 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:17,307 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:17,307 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:17,307 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,307 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:17,309 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,309 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:17,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:17,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:17,309 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,309 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:17,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:17,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:17,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:17,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:17,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:17,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:17,310 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:17,310 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:17,310 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:17,310 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:17,310 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:17,310 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:17,311 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:17,311 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:17,311 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:17,311 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:17,311 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:17,311 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,311 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:17,311 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:17,312 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,312 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:17,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:17,312 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,312 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:17,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:17,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:17,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:17,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:17,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:17,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c1a71d[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:17,314 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:17,314 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,315 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,315 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:17,315 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:17,315 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:17,315 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,315 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:17,319 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,319 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:17,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:17,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:17,319 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:17,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:17,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:17,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:17,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:17,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:17,324 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:17,324 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:17,324 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:17,324 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:17,324 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:17,326 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,326 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,326 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:17,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:17,327 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/42414bc697ad4899bbd86cdab462c9fc 2023-04-18 18:54:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:17,327 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:17,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:17,330 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,333 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/79acfe2c6e854ea2a43f219bb6faf9f0 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/79acfe2c6e854ea2a43f219bb6faf9f0 2023-04-18 18:54:17,335 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:17,335 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:17,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:17,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:17,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:17,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:17,338 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:17,338 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,338 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:17,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:17,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:17,338 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:17,338 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,338 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,338 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:17,338 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,339 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:17,339 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:17,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:17,342 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/79acfe2c6e854ea2a43f219bb6faf9f0, entries=200, sequenceid=474, filesize=32.3 K 2023-04-18 18:54:17,344 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/894b8e318d214f18b9bf0ed96bc87077 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/894b8e318d214f18b9bf0ed96bc87077 2023-04-18 18:54:17,349 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/894b8e318d214f18b9bf0ed96bc87077, entries=150, sequenceid=474, filesize=11.9 K 2023-04-18 18:54:17,350 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/42414bc697ad4899bbd86cdab462c9fc as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/42414bc697ad4899bbd86cdab462c9fc 2023-04-18 18:54:17,355 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/42414bc697ad4899bbd86cdab462c9fc, entries=150, sequenceid=474, filesize=11.9 K 2023-04-18 18:54:17,356 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for cbf5abd278bb2be3a248e6824371aa01 in 1011ms, sequenceid=474, compaction requested=false 2023-04-18 18:54:17,356 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:17,439 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:17,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:17,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:17,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:17,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:17,445 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:17,445 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:17,445 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:17,445 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:17,445 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,445 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:17,447 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:17,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:17,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:17,447 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:17,448 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,448 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:17,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:17,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:17,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:17,451 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:17,451 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:17,451 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:17,457 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:17,457 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:17,457 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:17,457 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:17,457 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 18:54:17,457 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:17,458 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:17,458 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:17,458 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:17,458 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:17,458 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:17,500 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:17,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:17,524 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041850a946cb3c3d45f18d6edc73aa9525e2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041850a946cb3c3d45f18d6edc73aa9525e2 2023-04-18 18:54:17,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:17,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50484 deadline: 1681844117530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:17,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844117532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:17,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844117533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,551 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=490, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/be29c06fbf1f4df9b966a68b73d54836 2023-04-18 18:54:17,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:17,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844117601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:17,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50520 deadline: 1681844117608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:17,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844117641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:17,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844117641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:17,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844117644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:17,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:17,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844117846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844117846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:17,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844117850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:17,994 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/653b8766284d436dbc1970b8448bdecb 2023-04-18 18:54:18,012 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/a4f91e1665c040b1afa572d2100a98f0 2023-04-18 18:54:18,025 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/be29c06fbf1f4df9b966a68b73d54836 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/be29c06fbf1f4df9b966a68b73d54836 2023-04-18 18:54:18,030 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/be29c06fbf1f4df9b966a68b73d54836, entries=150, sequenceid=490, filesize=25.4 K 2023-04-18 18:54:18,031 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/653b8766284d436dbc1970b8448bdecb as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/653b8766284d436dbc1970b8448bdecb 2023-04-18 18:54:18,039 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/653b8766284d436dbc1970b8448bdecb, entries=150, sequenceid=490, filesize=11.9 K 2023-04-18 18:54:18,041 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/a4f91e1665c040b1afa572d2100a98f0 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/a4f91e1665c040b1afa572d2100a98f0 2023-04-18 18:54:18,046 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/a4f91e1665c040b1afa572d2100a98f0, entries=150, sequenceid=490, filesize=11.9 K 2023-04-18 18:54:18,048 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for cbf5abd278bb2be3a248e6824371aa01 in 591ms, sequenceid=490, compaction requested=true 2023-04-18 18:54:18,048 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:18,048 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:18,048 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:18,048 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:18,048 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,048 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:18,048 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:18,051 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,051 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:18,052 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,052 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:18,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:18,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:18,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2adb6745[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:18,055 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:18,055 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,057 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,057 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:18,057 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:18,057 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:18,057 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,057 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:18,058 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,058 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:18,058 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:18,058 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:18,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:18,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:18,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:18,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:18,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:18,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:18,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:18,062 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:18,063 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:18,063 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:18,063 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:18,063 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:18,070 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,070 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:18,070 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,072 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:18,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:18,074 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,079 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,079 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:18,079 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:18,079 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,079 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:18,080 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,079 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:18,079 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:18,080 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,080 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:18,080 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:18,080 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:18,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:18,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:18,155 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 18:54:18,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844057523, max timestamp=1681844058154Num uniques 859; 2023-04-18 18:54:18,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:54:18,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844057523, max timestamp=1681844058154Num uniques 859; 2023-04-18 18:54:18,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:54:18,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844057523, max timestamp=1681844058154Num uniques 859; 2023-04-18 18:54:18,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:54:18,157 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:18,157 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:18,157 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:18,157 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:18,157 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:18,157 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:18,180 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:18,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:18,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681844058169, max timestamp=1681844058181Num uniques 859; 2023-04-18 18:54:18,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681844058169, max timestamp=1681844058181Num uniques 859; 2023-04-18 18:54:18,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-18 18:54:18,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-18 18:54:18,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681844058169, max timestamp=1681844058181Num uniques 859; 2023-04-18 18:54:18,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-18 18:54:18,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:18,188 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041820da66c283c84ae89b405e7110c31bca to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041820da66c283c84ae89b405e7110c31bca 2023-04-18 18:54:18,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:18,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:18,192 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:18,192 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:18,192 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:18,193 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:18,194 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,194 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,195 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:18,195 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:18,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,196 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,196 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:18,196 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=511, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/2e3c0c072c1645b4a8547b46b6d21c7a 2023-04-18 18:54:18,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:18,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:18,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:18,197 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:18,197 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:18,198 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:18,198 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:18,198 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:18,198 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:18,198 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:18,198 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:18,198 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:18,198 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:18,198 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:18,198 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,198 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:18,198 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:18,201 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,201 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:18,202 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,202 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:18,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:18,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37f8ee8[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:18,204 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:18,204 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,205 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,205 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:18,205 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:18,206 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:18,206 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,206 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:18,208 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:18,208 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:18,208 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:18,208 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:18,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:18,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:18,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,216 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=511 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/b74de95db2794a98acc34eb32b22d1ef 2023-04-18 18:54:18,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:18,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:18,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844118214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:18,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844118215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:18,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844118215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:18,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:18,219 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:18,219 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:18,220 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:18,220 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:18,220 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:18,221 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,221 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:18,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,221 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:18,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:18,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:18,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:18,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:18,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,223 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:18,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,226 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:18,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,230 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,230 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:18,230 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:18,230 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:18,230 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:18,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:18,230 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:18,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:18,230 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,231 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,231 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,231 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:18,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:18,231 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:18,231 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:18,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:18,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844118319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:18,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844118319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:18,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844118320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,331 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:18,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:18,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:18,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:18,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:18,334 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:18,334 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:18,334 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:18,334 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:18,335 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,335 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,336 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:18,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:18,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:18,337 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,337 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,337 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:18,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:18,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:18,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:18,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:18,338 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:18,338 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:18,338 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:18,340 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:18,340 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:18,340 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:18,340 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:18,340 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:18,340 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:18,340 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:18,340 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:18,340 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,340 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:18,340 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:18,342 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,342 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:18,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:18,342 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,342 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:18,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:18,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:18,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:18,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:18,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49a3199b[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:18,344 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:18,344 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,346 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,346 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,346 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:18,346 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:18,346 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:18,346 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:18,349 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:18,350 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:18,350 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:18,350 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:18,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:18,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:18,353 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:18,353 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:18,353 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:18,354 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:18,354 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:18,356 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,356 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,356 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:18,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:18,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:18,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:18,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:18,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:18,356 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:18,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,358 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:18,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,368 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,368 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,368 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:18,368 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:18,368 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:18,368 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:18,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:18,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:18,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:18,368 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,368 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,368 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,368 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:18,368 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:18,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:18,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:18,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:18,368 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,469 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:18,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:18,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:18,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:18,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:18,473 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:18,473 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:18,473 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:18,473 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:18,473 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,473 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,476 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,476 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:18,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:18,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:18,476 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,476 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:18,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:18,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,477 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:18,477 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:18,477 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:18,478 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:18,478 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:18,478 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:18,478 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:18,478 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:18,478 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:18,478 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:18,478 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:18,478 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:18,478 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:18,478 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:18,478 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,478 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:18,478 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:18,480 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,480 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:18,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:18,481 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,481 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:18,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:18,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:18,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:18,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:18,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45d6a6fd[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:18,483 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:18,483 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,484 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,484 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:18,484 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:18,484 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:18,484 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,484 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:18,486 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:18,486 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:18,486 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,486 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:18,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:18,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:18,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:18,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:18,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:18,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:18,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:18,490 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:18,490 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:18,490 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:18,490 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:18,490 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:18,493 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,493 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,493 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:18,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:18,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:18,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:18,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:18,494 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:18,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:18,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,496 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:18,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,499 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,499 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,499 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:18,499 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:18,499 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:18,499 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:18,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:18,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:18,499 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,500 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:18,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:18,500 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:18,500 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,500 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,500 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:18,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:18,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:18,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844118530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:18,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844118531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:18,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844118538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,600 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:18,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:18,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:18,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:18,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:18,603 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:18,604 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:18,604 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:18,604 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:18,605 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,605 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,607 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:18,607 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:18,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:18,607 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,607 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:18,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:18,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:18,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:18,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:18,612 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:18,612 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:18,612 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:18,612 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:18,612 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:18,612 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:18,612 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:18,613 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:18,613 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:18,613 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:18,613 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:18,613 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,613 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:18,613 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:18,615 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,615 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:18,616 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,616 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:18,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:18,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:18,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:18,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:18,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34c6104d[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:18,618 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:18,618 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,619 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,619 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:18,620 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:18,620 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:18,621 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:18,621 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,621 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:18,621 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:18,621 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:18,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:18,622 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:18,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:18,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:18,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:18,622 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:18,623 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,623 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,623 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:18,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:18,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:18,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:18,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:18,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,625 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:18,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:18,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:18,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:18,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,630 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=511 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/0cc3d3b3525d4098ad1e2b52c0552bb6 2023-04-18 18:54:18,631 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,634 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:18,634 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:18,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:18,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:18,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:18,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:18,634 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:18,634 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,634 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,634 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,634 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:18,634 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:18,634 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,635 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,635 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:18,635 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:18,636 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/2e3c0c072c1645b4a8547b46b6d21c7a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/2e3c0c072c1645b4a8547b46b6d21c7a 2023-04-18 18:54:18,640 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/2e3c0c072c1645b4a8547b46b6d21c7a, entries=250, sequenceid=511, filesize=39.2 K 2023-04-18 18:54:18,641 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/b74de95db2794a98acc34eb32b22d1ef as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/b74de95db2794a98acc34eb32b22d1ef 2023-04-18 18:54:18,646 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/b74de95db2794a98acc34eb32b22d1ef, entries=150, sequenceid=511, filesize=11.9 K 2023-04-18 18:54:18,647 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/0cc3d3b3525d4098ad1e2b52c0552bb6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/0cc3d3b3525d4098ad1e2b52c0552bb6 2023-04-18 18:54:18,652 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/0cc3d3b3525d4098ad1e2b52c0552bb6, entries=150, sequenceid=511, filesize=11.9 K 2023-04-18 18:54:18,653 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~81.42 KB/83376, currentSize=161.02 KB/164880 for cbf5abd278bb2be3a248e6824371aa01 in 499ms, sequenceid=511, compaction requested=true 2023-04-18 18:54:18,653 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:18,654 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:18,654 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:54:18,654 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:54:18,654 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:18,654 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:18,656 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48916 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:54:18,656 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 125480 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:54:18,656 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): cbf5abd278bb2be3a248e6824371aa01/B is initiating minor compaction (all files) 2023-04-18 18:54:18,656 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): cbf5abd278bb2be3a248e6824371aa01/A is initiating minor compaction (all files) 2023-04-18 18:54:18,656 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cbf5abd278bb2be3a248e6824371aa01/B in TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:18,656 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cbf5abd278bb2be3a248e6824371aa01/A in TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:18,656 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/176501c0f2d040fc8f0ee73d8588f1b3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/894b8e318d214f18b9bf0ed96bc87077, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/653b8766284d436dbc1970b8448bdecb, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/b74de95db2794a98acc34eb32b22d1ef] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp, totalSize=47.8 K 2023-04-18 18:54:18,656 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/97234ea094904e52aba1596f22b2f652, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/79acfe2c6e854ea2a43f219bb6faf9f0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/be29c06fbf1f4df9b966a68b73d54836, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/2e3c0c072c1645b4a8547b46b6d21c7a] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp, totalSize=122.5 K 2023-04-18 18:54:18,656 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 176501c0f2d040fc8f0ee73d8588f1b3, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681844055556 2023-04-18 18:54:18,656 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 97234ea094904e52aba1596f22b2f652, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681844055556 2023-04-18 18:54:18,657 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 894b8e318d214f18b9bf0ed96bc87077, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681844055696 2023-04-18 18:54:18,657 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 79acfe2c6e854ea2a43f219bb6faf9f0, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681844055696 2023-04-18 18:54:18,657 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 653b8766284d436dbc1970b8448bdecb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1681844056360 2023-04-18 18:54:18,657 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting be29c06fbf1f4df9b966a68b73d54836, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1681844056360 2023-04-18 18:54:18,658 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b74de95db2794a98acc34eb32b22d1ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1681844057530 2023-04-18 18:54:18,658 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 2e3c0c072c1645b4a8547b46b6d21c7a, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1681844057523 2023-04-18 18:54:18,674 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): cbf5abd278bb2be3a248e6824371aa01#B#compaction#140 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:18,707 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): cbf5abd278bb2be3a248e6824371aa01#A#compaction#141 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:18,735 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:18,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:18,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:18,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:18,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:18,739 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:18,739 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:18,740 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:18,740 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:18,740 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,740 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,742 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,742 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:18,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:18,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:18,742 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,742 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:18,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:18,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:18,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:18,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:18,743 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:18,743 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:18,743 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:18,743 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:18,743 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:18,744 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:18,744 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:18,744 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:18,744 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:18,744 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=161.02 KB heapSize=445.92 KB 2023-04-18 18:54:18,744 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:18,744 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:18,744 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:18,745 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:18,745 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:18,745 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:18,768 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418d652429c2a7b45169e1ea231a55d8898 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418d652429c2a7b45169e1ea231a55d8898 2023-04-18 18:54:18,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:18,835 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:18,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:18,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844118841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:18,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844118842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:18,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844118843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:18,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844118946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:18,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:18,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844118946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:19,110 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/2dbc95602be54157a186710e92fbb49c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/2dbc95602be54157a186710e92fbb49c 2023-04-18 18:54:19,116 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cbf5abd278bb2be3a248e6824371aa01/B of cbf5abd278bb2be3a248e6824371aa01 into 2dbc95602be54157a186710e92fbb49c(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:19,116 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:19,116 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., storeName=cbf5abd278bb2be3a248e6824371aa01/B, priority=12, startTime=1681844058654; duration=0sec 2023-04-18 18:54:19,117 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:19,117 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:54:19,119 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48916 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:54:19,119 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): cbf5abd278bb2be3a248e6824371aa01/C is initiating minor compaction (all files) 2023-04-18 18:54:19,119 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cbf5abd278bb2be3a248e6824371aa01/C in TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:19,119 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/20f130deb76344ad9f0bf0e979848848, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/42414bc697ad4899bbd86cdab462c9fc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/a4f91e1665c040b1afa572d2100a98f0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/0cc3d3b3525d4098ad1e2b52c0552bb6] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp, totalSize=47.8 K 2023-04-18 18:54:19,120 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 20f130deb76344ad9f0bf0e979848848, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681844055556 2023-04-18 18:54:19,120 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 42414bc697ad4899bbd86cdab462c9fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681844055696 2023-04-18 18:54:19,121 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting a4f91e1665c040b1afa572d2100a98f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1681844056360 2023-04-18 18:54:19,121 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 0cc3d3b3525d4098ad1e2b52c0552bb6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1681844057530 2023-04-18 18:54:19,135 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): cbf5abd278bb2be3a248e6824371aa01#C#compaction#143 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:19,141 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/3357de25dcc2400aab3824afee4f5a6b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/3357de25dcc2400aab3824afee4f5a6b 2023-04-18 18:54:19,151 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cbf5abd278bb2be3a248e6824371aa01/A of cbf5abd278bb2be3a248e6824371aa01 into 3357de25dcc2400aab3824afee4f5a6b(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:19,152 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:19,152 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., storeName=cbf5abd278bb2be3a248e6824371aa01/A, priority=12, startTime=1681844058653; duration=0sec 2023-04-18 18:54:19,152 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:19,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:19,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844119149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:19,155 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/577d83a5163d489d960cc06957720521 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/577d83a5163d489d960cc06957720521 2023-04-18 18:54:19,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:19,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844119150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:19,164 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cbf5abd278bb2be3a248e6824371aa01/C of cbf5abd278bb2be3a248e6824371aa01 into 577d83a5163d489d960cc06957720521(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:19,164 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:19,164 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., storeName=cbf5abd278bb2be3a248e6824371aa01/C, priority=12, startTime=1681844058654; duration=0sec 2023-04-18 18:54:19,164 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:19,178 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=548, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/fb95d579ce9f4593be4ba7426e8d89de 2023-04-18 18:54:19,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:19,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844119348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:19,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:19,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844119457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:19,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:19,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844119457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:19,599 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/ee2b2ffdfa4144a995756dd2f012765b 2023-04-18 18:54:19,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:19,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844119615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:19,617 DEBUG [Thread-855] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4136 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:54:19,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:19,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50520 deadline: 1681844119628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:19,630 DEBUG [Thread-847] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:54:19,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:19,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844119960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:19,962 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:19,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50514 deadline: 1681844119961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,012 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/7ad8c3b0369e43a290f387d76fd30498 2023-04-18 18:54:20,017 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/fb95d579ce9f4593be4ba7426e8d89de as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/fb95d579ce9f4593be4ba7426e8d89de 2023-04-18 18:54:20,021 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/fb95d579ce9f4593be4ba7426e8d89de, entries=150, sequenceid=548, filesize=25.4 K 2023-04-18 18:54:20,022 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/ee2b2ffdfa4144a995756dd2f012765b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/ee2b2ffdfa4144a995756dd2f012765b 2023-04-18 18:54:20,029 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/ee2b2ffdfa4144a995756dd2f012765b, entries=150, sequenceid=548, filesize=11.9 K 2023-04-18 18:54:20,030 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/7ad8c3b0369e43a290f387d76fd30498 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/7ad8c3b0369e43a290f387d76fd30498 2023-04-18 18:54:20,042 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/7ad8c3b0369e43a290f387d76fd30498, entries=150, sequenceid=548, filesize=11.9 K 2023-04-18 18:54:20,043 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~445.88 KB/456576, currentSize=26.84 KB/27480 for cbf5abd278bb2be3a248e6824371aa01 in 1299ms, sequenceid=548, compaction requested=false 2023-04-18 18:54:20,043 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:20,043 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:20,044 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:20,044 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:20,044 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,044 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:20,044 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:20,045 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,045 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:20,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:20,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:20,046 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:20,046 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:20,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:20,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:20,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:20,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:20,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:20,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b31d0f3[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:20,048 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:20,048 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:20,049 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:20,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:20,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:20,049 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:20,049 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:20,049 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:20,049 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,049 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:20,052 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:20,052 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:20,052 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:20,052 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:20,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:20,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:20,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:20,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:20,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:20,055 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:20,055 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:20,055 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:20,055 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:20,055 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:20,057 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:20,057 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:20,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:20,057 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:20,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:20,058 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:20,059 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:20,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,062 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:20,062 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:20,062 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,062 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:20,062 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:20,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:20,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:20,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:20,062 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,062 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:20,062 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:20,062 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:20,062 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:20,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:20,062 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:20,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:20,062 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:20,162 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:20,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:20,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:20,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:20,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:20,166 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:20,166 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:20,166 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:20,166 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:20,166 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:20,167 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:20,168 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:20,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:20,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:20,168 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,169 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,169 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:20,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:20,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:20,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:20,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:20,170 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:20,170 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:20,170 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:20,170 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:20,170 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:20,170 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:20,170 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:20,171 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-18 18:54:20,171 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:20,171 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:20,171 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:20,171 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:20,171 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:20,171 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:20,195 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304186b6c2a799fac4da1a23a922f48d587ac to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304186b6c2a799fac4da1a23a922f48d587ac 2023-04-18 18:54:20,209 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=558, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/e7a20fd9aea344ac8513df25797352a9 2023-04-18 18:54:20,262 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=558 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/67be30e6125c4106be54fb0691433b7a 2023-04-18 18:54:20,280 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=558 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/276135bd3cf744f8966f7c2dccc1cd55 2023-04-18 18:54:20,288 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/e7a20fd9aea344ac8513df25797352a9 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/e7a20fd9aea344ac8513df25797352a9 2023-04-18 18:54:20,293 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/e7a20fd9aea344ac8513df25797352a9, entries=100, sequenceid=558, filesize=18.6 K 2023-04-18 18:54:20,294 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/67be30e6125c4106be54fb0691433b7a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/67be30e6125c4106be54fb0691433b7a 2023-04-18 18:54:20,300 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/67be30e6125c4106be54fb0691433b7a, entries=100, sequenceid=558, filesize=9.5 K 2023-04-18 18:54:20,302 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/276135bd3cf744f8966f7c2dccc1cd55 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/276135bd3cf744f8966f7c2dccc1cd55 2023-04-18 18:54:20,307 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/276135bd3cf744f8966f7c2dccc1cd55, entries=100, sequenceid=558, filesize=9.5 K 2023-04-18 18:54:20,308 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for cbf5abd278bb2be3a248e6824371aa01 in 137ms, sequenceid=558, compaction requested=true 2023-04-18 18:54:20,308 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:20,308 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:20,308 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:20,308 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:20,308 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,310 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:20,310 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:20,313 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,313 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:20,313 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:20,313 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:20,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:20,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a0f67cf[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:20,315 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:20,315 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:20,316 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:20,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:20,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:20,317 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:20,317 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:20,317 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:20,317 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,317 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:20,318 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:20,318 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:20,318 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,318 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:20,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:20,326 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:20,327 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:20,327 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:20,327 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:20,327 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:20,329 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:20,329 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:20,329 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:20,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:20,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:20,330 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:20,332 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,338 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,339 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:20,339 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:20,339 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,339 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:20,339 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:20,339 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:20,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:20,339 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:20,339 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:20,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:20,339 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:20,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:20,340 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:20,340 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:20,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:20,440 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:20,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:20,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:20,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:20,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:20,444 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:20,444 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:20,444 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:20,444 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:20,444 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:20,444 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:20,445 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:20,445 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:20,446 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:20,446 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:20,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:20,447 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:20,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:20,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:20,448 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:20,448 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:20,448 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:20,448 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:20,448 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:20,448 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:20,449 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:20,449 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-18 18:54:20,449 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:20,449 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:20,450 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:20,450 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:20,450 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:20,450 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:20,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:20,472 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:20,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844060491, max timestamp=1681844060533Num uniques 859; 2023-04-18 18:54:20,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:54:20,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844060491, max timestamp=1681844060533Num uniques 859; 2023-04-18 18:54:20,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:54:20,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844060491, max timestamp=1681844060533Num uniques 859; 2023-04-18 18:54:20,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:54:20,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681844060567, max timestamp=1681844060583Num uniques 1068; 2023-04-18 18:54:20,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681844060567, max timestamp=1681844060583Num uniques 1068; 2023-04-18 18:54:20,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-18 18:54:20,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-18 18:54:20,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681844060567, max timestamp=1681844060583Num uniques 1068; 2023-04-18 18:54:20,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-18 18:54:20,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681844060595, max timestamp=1681844060641Num uniques 1077; 2023-04-18 18:54:20,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-18 18:54:20,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681844060595, max timestamp=1681844060641Num uniques 1077; 2023-04-18 18:54:20,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681844060595, max timestamp=1681844060641Num uniques 1077; 2023-04-18 18:54:20,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-18 18:54:20,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-18 18:54:20,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681844060690, max timestamp=1681844060710Num uniques 1036; 2023-04-18 18:54:20,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-18 18:54:20,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681844060690, max timestamp=1681844060710Num uniques 1036; 2023-04-18 18:54:20,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681844060690, max timestamp=1681844060710Num uniques 1036; 2023-04-18 18:54:20,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-18 18:54:20,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-18 18:54:20,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681844060739, max timestamp=1681844060761Num uniques 1095; 2023-04-18 18:54:20,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-18 18:54:20,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681844060739, max timestamp=1681844060761Num uniques 1095; 2023-04-18 18:54:20,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-18 18:54:20,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681844060739, max timestamp=1681844060761Num uniques 1095; 2023-04-18 18:54:20,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-18 18:54:20,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681844060791, max timestamp=1681844060829Num uniques 1054; 2023-04-18 18:54:20,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681844060791, max timestamp=1681844060829Num uniques 1054; 2023-04-18 18:54:20,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-18 18:54:20,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-18 18:54:20,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681844060791, max timestamp=1681844060829Num uniques 1054; 2023-04-18 18:54:20,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-18 18:54:20,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681844060852, max timestamp=1681844060886Num uniques 1013; 2023-04-18 18:54:20,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-18 18:54:20,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681844060852, max timestamp=1681844060886Num uniques 1013; 2023-04-18 18:54:20,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-18 18:54:20,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681844060852, max timestamp=1681844060886Num uniques 1013; 2023-04-18 18:54:20,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-18 18:54:20,894 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418368ed5f857714b8f96c1dc1691c8cf82 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418368ed5f857714b8f96c1dc1691c8cf82 2023-04-18 18:54:20,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681844060932, max timestamp=1681844060955Num uniques 1022; 2023-04-18 18:54:20,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681844060932, max timestamp=1681844060955Num uniques 1022; 2023-04-18 18:54:20,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681844060932, max timestamp=1681844060955Num uniques 1022; 2023-04-18 18:54:20,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-18 18:54:20,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-18 18:54:20,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-18 18:54:20,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681844060975, max timestamp=1681844060985Num uniques 1031; 2023-04-18 18:54:20,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-18 18:54:20,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681844060975, max timestamp=1681844060985Num uniques 1031; 2023-04-18 18:54:20,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-18 18:54:20,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8282, totalHeapSize=19064, min timestamp=1681844060975, max timestamp=1681844060985Num uniques 1031; 2023-04-18 18:54:20,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47212, new segment data size=8282, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-18 18:54:21,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1681844061000, max timestamp=1681844061010Num uniques 1040; 2023-04-18 18:54:21,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-18 18:54:21,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1681844061000, max timestamp=1681844061010Num uniques 1040; 2023-04-18 18:54:21,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-18 18:54:21,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1681844061000, max timestamp=1681844061010Num uniques 1040; 2023-04-18 18:54:21,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-18 18:54:21,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681844061023, max timestamp=1681844061034Num uniques 1149; 2023-04-18 18:54:21,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681844061023, max timestamp=1681844061034Num uniques 1149; 2023-04-18 18:54:21,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681844061023, max timestamp=1681844061034Num uniques 1149; 2023-04-18 18:54:21,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=9114, suffix heap size=147464,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=199 2023-04-18 18:54:21,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=9114, suffix heap size=147464,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=199 2023-04-18 18:54:21,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=9114, suffix heap size=147464,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=199 2023-04-18 18:54:21,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=308, cellSize=14100, totalHeapSize=32272, min timestamp=1681844061049, max timestamp=1681844061065Num uniques 1058; 2023-04-18 18:54:21,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=14100, suffix heap size=138000,new segment heap size=32272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=308 2023-04-18 18:54:21,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=308, cellSize=14100, totalHeapSize=32272, min timestamp=1681844061049, max timestamp=1681844061065Num uniques 1058; 2023-04-18 18:54:21,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=14100, suffix heap size=138000,new segment heap size=32272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=308 2023-04-18 18:54:21,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=308, cellSize=14100, totalHeapSize=32272, min timestamp=1681844061049, max timestamp=1681844061065Num uniques 1058; 2023-04-18 18:54:21,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=14100, suffix heap size=138000,new segment heap size=32272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=308 2023-04-18 18:54:21,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=417, cellSize=19092, totalHeapSize=43608, min timestamp=1681844061077, max timestamp=1681844061088Num uniques 1167; 2023-04-18 18:54:21,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=417, cellSize=19092, totalHeapSize=43608, min timestamp=1681844061077, max timestamp=1681844061088Num uniques 1167; 2023-04-18 18:54:21,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53442, new segment data size=19092, suffix heap size=149336,new segment heap size=43608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1167, new segment cells count=417 2023-04-18 18:54:21,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=417, cellSize=19092, totalHeapSize=43608, min timestamp=1681844061077, max timestamp=1681844061088Num uniques 1167; 2023-04-18 18:54:21,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53442, new segment data size=19092, suffix heap size=149336,new segment heap size=43608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1167, new segment cells count=417 2023-04-18 18:54:21,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53442, new segment data size=19092, suffix heap size=149336,new segment heap size=43608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1167, new segment cells count=417 2023-04-18 18:54:21,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10336, totalHeapSize=23744, min timestamp=1681844061103, max timestamp=1681844061112Num uniques 1276; 2023-04-18 18:54:21,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=58426, new segment data size=10336, suffix heap size=160672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1276, new segment cells count=226 2023-04-18 18:54:21,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10336, totalHeapSize=23744, min timestamp=1681844061103, max timestamp=1681844061112Num uniques 1276; 2023-04-18 18:54:21,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=58426, new segment data size=10336, suffix heap size=160672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1276, new segment cells count=226 2023-04-18 18:54:21,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10336, totalHeapSize=23744, min timestamp=1681844061103, max timestamp=1681844061112Num uniques 1276; 2023-04-18 18:54:21,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=58426, new segment data size=10336, suffix heap size=160672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1276, new segment cells count=226 2023-04-18 18:54:21,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1681844061150, max timestamp=1681844061160Num uniques 1085; 2023-04-18 18:54:21,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=13050, suffix heap size=140808,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=285 2023-04-18 18:54:21,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1681844061150, max timestamp=1681844061160Num uniques 1085; 2023-04-18 18:54:21,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1681844061150, max timestamp=1681844061160Num uniques 1085; 2023-04-18 18:54:21,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=13050, suffix heap size=140808,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=285 2023-04-18 18:54:21,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=13050, suffix heap size=140808,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=285 2023-04-18 18:54:21,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681844061178, max timestamp=1681844061187Num uniques 1144; 2023-04-18 18:54:21,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=11174, suffix heap size=146944,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=244 2023-04-18 18:54:21,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681844061178, max timestamp=1681844061187Num uniques 1144; 2023-04-18 18:54:21,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=11174, suffix heap size=146944,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=244 2023-04-18 18:54:21,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681844061178, max timestamp=1681844061187Num uniques 1144; 2023-04-18 18:54:21,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=11174, suffix heap size=146944,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=244 2023-04-18 18:54:21,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681844061226, max timestamp=1681844061230Num uniques 1103; 2023-04-18 18:54:21,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-04-18 18:54:21,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681844061226, max timestamp=1681844061230Num uniques 1103; 2023-04-18 18:54:21,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681844061226, max timestamp=1681844061230Num uniques 1103; 2023-04-18 18:54:21,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-04-18 18:54:21,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-04-18 18:54:21,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1681844061246, max timestamp=1681844061256Num uniques 1012; 2023-04-18 18:54:21,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1681844061246, max timestamp=1681844061256Num uniques 1012; 2023-04-18 18:54:21,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=11992, suffix heap size=133216,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=262 2023-04-18 18:54:21,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1681844061246, max timestamp=1681844061256Num uniques 1012; 2023-04-18 18:54:21,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=11992, suffix heap size=133216,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=262 2023-04-18 18:54:21,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=11992, suffix heap size=133216,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=262 2023-04-18 18:54:21,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=371, cellSize=16986, totalHeapSize=38824, min timestamp=1681844061255, max timestamp=1681844061277Num uniques 1121; 2023-04-18 18:54:21,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=371, cellSize=16986, totalHeapSize=38824, min timestamp=1681844061255, max timestamp=1681844061277Num uniques 1121; 2023-04-18 18:54:21,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=16986, suffix heap size=144552,new segment heap size=38824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=371 2023-04-18 18:54:21,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=16986, suffix heap size=144552,new segment heap size=38824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=371 2023-04-18 18:54:21,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=371, cellSize=16986, totalHeapSize=38824, min timestamp=1681844061255, max timestamp=1681844061277Num uniques 1121; 2023-04-18 18:54:21,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=16986, suffix heap size=144552,new segment heap size=38824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=371 2023-04-18 18:54:21,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1681844061292, max timestamp=1681844061297Num uniques 1230; 2023-04-18 18:54:21,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=56330, new segment data size=12820, suffix heap size=155888,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1230, new segment cells count=280 2023-04-18 18:54:21,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1681844061292, max timestamp=1681844061297Num uniques 1230; 2023-04-18 18:54:21,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1681844061292, max timestamp=1681844061297Num uniques 1230; 2023-04-18 18:54:21,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=56330, new segment data size=12820, suffix heap size=155888,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1230, new segment cells count=280 2023-04-18 18:54:21,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=56330, new segment data size=12820, suffix heap size=155888,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1230, new segment cells count=280 2023-04-18 18:54:21,301 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=563, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/c81ada1ff9e64090906f93ce117e0f72 2023-04-18 18:54:21,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=539, cellSize=24684, totalHeapSize=56296, min timestamp=1681844061332, max timestamp=1681844061345Num uniques 1139; 2023-04-18 18:54:21,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=24684, suffix heap size=146424,new segment heap size=56296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=539 2023-04-18 18:54:21,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=539, cellSize=24684, totalHeapSize=56296, min timestamp=1681844061332, max timestamp=1681844061345Num uniques 1139; 2023-04-18 18:54:21,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=24684, suffix heap size=146424,new segment heap size=56296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=539 2023-04-18 18:54:21,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=539, cellSize=24684, totalHeapSize=56296, min timestamp=1681844061332, max timestamp=1681844061345Num uniques 1139; 2023-04-18 18:54:21,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=24684, suffix heap size=146424,new segment heap size=56296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=539 2023-04-18 18:54:21,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:21,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844121368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:21,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844121369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:21,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 443 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844121370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:21,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844121473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:21,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 445 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844121473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:21,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844121473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:21,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844121677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:21,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844121677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:21,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 447 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844121678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,714 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=563 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/0021b1b233394ded90cdf8792287c6f3 2023-04-18 18:54:21,739 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=563 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/8d202aa3c1e04bb4a23709fbaf8b4184 2023-04-18 18:54:21,755 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/c81ada1ff9e64090906f93ce117e0f72 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/c81ada1ff9e64090906f93ce117e0f72 2023-04-18 18:54:21,760 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/c81ada1ff9e64090906f93ce117e0f72, entries=100, sequenceid=563, filesize=18.6 K 2023-04-18 18:54:21,762 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/0021b1b233394ded90cdf8792287c6f3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/0021b1b233394ded90cdf8792287c6f3 2023-04-18 18:54:21,771 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/0021b1b233394ded90cdf8792287c6f3, entries=100, sequenceid=563, filesize=9.5 K 2023-04-18 18:54:21,772 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/8d202aa3c1e04bb4a23709fbaf8b4184 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/8d202aa3c1e04bb4a23709fbaf8b4184 2023-04-18 18:54:21,778 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/8d202aa3c1e04bb4a23709fbaf8b4184, entries=100, sequenceid=563, filesize=9.5 K 2023-04-18 18:54:21,779 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=181.14 KB/185490 for cbf5abd278bb2be3a248e6824371aa01 in 1330ms, sequenceid=563, compaction requested=true 2023-04-18 18:54:21,779 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:21,779 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:21,779 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:21,779 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:21,779 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,779 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:21,779 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:21,781 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,781 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:21,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:21,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:21,781 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:21,781 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:21,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:21,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:21,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:21,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:21,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:21,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a9edb3d[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:21,783 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:21,783 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:21,785 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:21,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:21,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:21,785 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:21,786 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:21,786 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,786 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:21,786 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:21,788 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:21,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:21,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:21,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:21,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:21,791 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:21,791 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:21,791 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:21,791 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:21,791 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:21,793 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:21,793 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:21,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:21,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:21,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:21,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:21,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:21,793 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:21,794 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:21,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:21,795 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:21,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,797 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,798 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:21,798 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:21,798 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,798 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:21,798 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:21,798 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:21,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:21,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:21,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:21,798 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:21,798 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:21,798 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:21,798 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:21,798 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:21,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:21,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:21,898 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:21,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:21,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:21,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:21,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:21,902 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:21,902 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:21,902 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:21,902 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:21,903 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:21,903 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:21,905 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:21,905 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:21,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:21,905 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,905 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:21,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:21,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:21,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:21,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:21,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:21,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:21,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:21,906 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:21,906 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:21,906 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:21,907 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:21,907 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:21,907 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:21,907 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:21,907 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=181.14 KB heapSize=488.81 KB 2023-04-18 18:54:21,907 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:21,907 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:21,908 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:21,908 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:21,908 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:21,908 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:21,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:21,987 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:21,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:21,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844121986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:21,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844121987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:21,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:21,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 450 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844121988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:22,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:22,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844122089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:22,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:22,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 452 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844122092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:22,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:22,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844122293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:22,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:22,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 454 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844122298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:22,327 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304184587a77d83a24cad8b61df6ad2793f2b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304184587a77d83a24cad8b61df6ad2793f2b 2023-04-18 18:54:22,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:22,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844122491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:22,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:22,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844122598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:22,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:22,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 456 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844122601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:22,734 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=943, memsize=60.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/ca26529115ea4844ac4099537708e73d 2023-04-18 18:54:23,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:23,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844123102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:23,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:23,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 458 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844123106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:23,146 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=943 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/62d845575eb548659248f3c82884665c 2023-04-18 18:54:23,158 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=943 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/65a2bb22879d4f20ac3ddfff949699c6 2023-04-18 18:54:23,164 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/ca26529115ea4844ac4099537708e73d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/ca26529115ea4844ac4099537708e73d 2023-04-18 18:54:23,168 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/ca26529115ea4844ac4099537708e73d, entries=150, sequenceid=943, filesize=25.4 K 2023-04-18 18:54:23,169 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/62d845575eb548659248f3c82884665c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/62d845575eb548659248f3c82884665c 2023-04-18 18:54:23,173 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/62d845575eb548659248f3c82884665c, entries=150, sequenceid=943, filesize=11.9 K 2023-04-18 18:54:23,174 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/65a2bb22879d4f20ac3ddfff949699c6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/65a2bb22879d4f20ac3ddfff949699c6 2023-04-18 18:54:23,178 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/65a2bb22879d4f20ac3ddfff949699c6, entries=150, sequenceid=943, filesize=11.9 K 2023-04-18 18:54:23,179 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~181.14 KB/185490, heapSize ~488.77 KB/500496, currentSize=13.42 KB/13740 for cbf5abd278bb2be3a248e6824371aa01 in 1272ms, sequenceid=943, compaction requested=true 2023-04-18 18:54:23,179 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:23,179 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:23,179 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:23,179 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:23,179 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:23,179 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:23,179 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:23,182 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:23,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:23,182 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:23,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:23,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:23,182 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:23,182 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:23,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:23,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:23,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:23,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:23,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:23,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:23,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6edefa5b[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:23,184 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:23,184 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:23,186 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:23,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:23,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:23,186 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:23,186 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:23,186 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:23,188 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:23,188 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:23,188 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:23,188 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:23,188 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:23,189 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:23,189 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:23,189 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:23,189 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:23,189 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:23,189 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:23,192 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:23,192 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:23,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:23,192 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:23,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:23,193 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:23,194 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:23,196 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:23,196 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:23,196 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:23,196 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:23,196 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:23,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:23,196 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:23,196 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:23,197 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:23,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:23,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:23,196 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:23,197 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:23,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:23,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:23,197 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:23,197 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:23,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:23,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:23,297 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:23,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:23,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:23,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:23,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:23,302 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:23,302 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:23,302 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:23,302 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:23,302 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:23,302 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:23,304 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:23,304 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:23,304 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:23,305 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:23,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:23,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:23,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:23,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:23,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:23,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:23,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:23,309 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:23,309 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:23,309 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:23,310 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:23,310 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:23,310 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:23,310 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:23,310 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-18 18:54:23,310 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:23,310 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:23,310 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:23,310 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:23,310 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:23,311 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:23,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:23,513 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:23,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844063570, max timestamp=1681844063582Num uniques 859; 2023-04-18 18:54:23,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:54:23,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844063570, max timestamp=1681844063582Num uniques 859; 2023-04-18 18:54:23,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844063570, max timestamp=1681844063582Num uniques 859; 2023-04-18 18:54:23,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:54:23,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:54:23,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681844063604, max timestamp=1681844063631Num uniques 1068; 2023-04-18 18:54:23,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-18 18:54:23,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681844063604, max timestamp=1681844063631Num uniques 1068; 2023-04-18 18:54:23,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-18 18:54:23,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681844063604, max timestamp=1681844063631Num uniques 1068; 2023-04-18 18:54:23,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-18 18:54:23,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681844063649, max timestamp=1681844063662Num uniques 1077; 2023-04-18 18:54:23,664 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-18 18:54:23,664 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681844063649, max timestamp=1681844063662Num uniques 1077; 2023-04-18 18:54:23,664 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681844063649, max timestamp=1681844063662Num uniques 1077; 2023-04-18 18:54:23,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-18 18:54:23,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-18 18:54:23,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681844063679, max timestamp=1681844063691Num uniques 1136; 2023-04-18 18:54:23,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-18 18:54:23,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681844063679, max timestamp=1681844063691Num uniques 1136; 2023-04-18 18:54:23,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681844063679, max timestamp=1681844063691Num uniques 1136; 2023-04-18 18:54:23,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-18 18:54:23,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-18 18:54:23,727 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418c5b520703f4a4021b7a3c065602dc350 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418c5b520703f4a4021b7a3c065602dc350 2023-04-18 18:54:23,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1681844063708, max timestamp=1681844063746Num uniques 1095; 2023-04-18 18:54:23,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1681844063708, max timestamp=1681844063746Num uniques 1095; 2023-04-18 18:54:23,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=18090, suffix heap size=141848,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=395 2023-04-18 18:54:23,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=18090, suffix heap size=141848,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=395 2023-04-18 18:54:23,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1681844063708, max timestamp=1681844063746Num uniques 1095; 2023-04-18 18:54:23,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=18090, suffix heap size=141848,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=395 2023-04-18 18:54:23,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=404, cellSize=18500, totalHeapSize=42256, min timestamp=1681844063754, max timestamp=1681844063776Num uniques 1254; 2023-04-18 18:54:23,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=404, cellSize=18500, totalHeapSize=42256, min timestamp=1681844063754, max timestamp=1681844063776Num uniques 1254; 2023-04-18 18:54:23,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=57430, new segment data size=18500, suffix heap size=158384,new segment heap size=42256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1254, new segment cells count=404 2023-04-18 18:54:23,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=57430, new segment data size=18500, suffix heap size=158384,new segment heap size=42256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1254, new segment cells count=404 2023-04-18 18:54:23,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=404, cellSize=18500, totalHeapSize=42256, min timestamp=1681844063754, max timestamp=1681844063776Num uniques 1254; 2023-04-18 18:54:23,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=57430, new segment data size=18500, suffix heap size=158384,new segment heap size=42256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1254, new segment cells count=404 2023-04-18 18:54:23,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1681844063790, max timestamp=1681844063798Num uniques 1263; 2023-04-18 18:54:23,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=57838, new segment data size=14328, suffix heap size=159320,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1263, new segment cells count=313 2023-04-18 18:54:23,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1681844063790, max timestamp=1681844063798Num uniques 1263; 2023-04-18 18:54:23,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1681844063790, max timestamp=1681844063798Num uniques 1263; 2023-04-18 18:54:23,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=57838, new segment data size=14328, suffix heap size=159320,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1263, new segment cells count=313 2023-04-18 18:54:23,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=57838, new segment data size=14328, suffix heap size=159320,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1263, new segment cells count=313 2023-04-18 18:54:23,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1681844063810, max timestamp=1681844063820Num uniques 1172; 2023-04-18 18:54:23,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1681844063810, max timestamp=1681844063820Num uniques 1172; 2023-04-18 18:54:23,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=14742, suffix heap size=149856,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=322 2023-04-18 18:54:23,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1681844063810, max timestamp=1681844063820Num uniques 1172; 2023-04-18 18:54:23,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=14742, suffix heap size=149856,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=322 2023-04-18 18:54:23,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=14742, suffix heap size=149856,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=322 2023-04-18 18:54:23,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=381, cellSize=17446, totalHeapSize=39864, min timestamp=1681844063854, max timestamp=1681844063867Num uniques 1181; 2023-04-18 18:54:23,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=17446, suffix heap size=150792,new segment heap size=39864 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=381 2023-04-18 18:54:23,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=381, cellSize=17446, totalHeapSize=39864, min timestamp=1681844063854, max timestamp=1681844063867Num uniques 1181; 2023-04-18 18:54:23,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=381, cellSize=17446, totalHeapSize=39864, min timestamp=1681844063854, max timestamp=1681844063867Num uniques 1181; 2023-04-18 18:54:23,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=17446, suffix heap size=150792,new segment heap size=39864 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=381 2023-04-18 18:54:23,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=17446, suffix heap size=150792,new segment heap size=39864 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=381 2023-04-18 18:54:23,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681844063906, max timestamp=1681844063915Num uniques 1240; 2023-04-18 18:54:23,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=56790, new segment data size=8700, suffix heap size=156928,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1240, new segment cells count=190 2023-04-18 18:54:23,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681844063906, max timestamp=1681844063915Num uniques 1240; 2023-04-18 18:54:23,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=56790, new segment data size=8700, suffix heap size=156928,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1240, new segment cells count=190 2023-04-18 18:54:23,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681844063906, max timestamp=1681844063915Num uniques 1240; 2023-04-18 18:54:23,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=56790, new segment data size=8700, suffix heap size=156928,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1240, new segment cells count=190 2023-04-18 18:54:23,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=449, cellSize=20564, totalHeapSize=46936, min timestamp=1681844063928, max timestamp=1681844063939Num uniques 1049; 2023-04-18 18:54:23,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=449, cellSize=20564, totalHeapSize=46936, min timestamp=1681844063928, max timestamp=1681844063939Num uniques 1049; 2023-04-18 18:54:23,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=20564, suffix heap size=137064,new segment heap size=46936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=449 2023-04-18 18:54:23,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=449, cellSize=20564, totalHeapSize=46936, min timestamp=1681844063928, max timestamp=1681844063939Num uniques 1049; 2023-04-18 18:54:23,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=20564, suffix heap size=137064,new segment heap size=46936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=449 2023-04-18 18:54:23,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=20564, suffix heap size=137064,new segment heap size=46936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=449 2023-04-18 18:54:23,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:23,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50520 deadline: 1681844123960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:23,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:23,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844123961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:23,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:23,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844123966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:24,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50520 deadline: 1681844124067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:24,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844124067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,073 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:24,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844124072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:24,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 460 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844124108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:24,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844124108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,134 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=948, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/076d998a3f474df6967624d83c17e1a2 2023-04-18 18:54:24,150 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=948 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/bc42c0e5139e4d49901d4f27345f0341 2023-04-18 18:54:24,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:24,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50520 deadline: 1681844124271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:24,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844124272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:24,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844124274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,568 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=948 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/97c3768a16a34c6dbeccfecd397bb7cd 2023-04-18 18:54:24,573 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/076d998a3f474df6967624d83c17e1a2 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/076d998a3f474df6967624d83c17e1a2 2023-04-18 18:54:24,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:24,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844124575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:24,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50520 deadline: 1681844124575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:24,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844124578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,579 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/076d998a3f474df6967624d83c17e1a2, entries=100, sequenceid=948, filesize=18.6 K 2023-04-18 18:54:24,580 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/bc42c0e5139e4d49901d4f27345f0341 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/bc42c0e5139e4d49901d4f27345f0341 2023-04-18 18:54:24,585 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/bc42c0e5139e4d49901d4f27345f0341, entries=100, sequenceid=948, filesize=9.5 K 2023-04-18 18:54:24,586 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/97c3768a16a34c6dbeccfecd397bb7cd as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/97c3768a16a34c6dbeccfecd397bb7cd 2023-04-18 18:54:24,590 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/97c3768a16a34c6dbeccfecd397bb7cd, entries=100, sequenceid=948, filesize=9.5 K 2023-04-18 18:54:24,591 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=174.43 KB/178620 for cbf5abd278bb2be3a248e6824371aa01 in 1281ms, sequenceid=948, compaction requested=true 2023-04-18 18:54:24,591 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:24,591 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:24,591 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:24,591 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:24,591 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,591 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:24,591 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:24,593 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:24,594 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:24,594 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:24,594 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:24,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:24,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:24,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:24,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:24,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:24,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:24,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b2d193b[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:24,596 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:24,596 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:24,599 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:24,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:24,599 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:24,599 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:24,599 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:24,599 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,599 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:24,600 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:24,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:24,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:24,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:24,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:24,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:24,603 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:24,603 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:24,603 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:24,603 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:24,603 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:24,605 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:24,605 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:24,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:24,605 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:24,605 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,607 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:24,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:24,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,609 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,609 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:24,609 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:24,609 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:24,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:24,609 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:24,609 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:24,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:24,609 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:24,609 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:24,609 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:24,609 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:24,610 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:24,610 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:24,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:24,710 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:24,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:24,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:24,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:24,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:24,713 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:24,713 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:24,713 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:24,713 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:24,714 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:24,714 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:24,716 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,716 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:24,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:24,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:24,716 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:24,716 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:24,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:24,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:24,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:24,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:24,717 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:24,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:24,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:24,718 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:24,718 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:24,718 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:24,718 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:24,718 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:24,719 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:24,719 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:24,719 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=174.43 KB heapSize=477.33 KB 2023-04-18 18:54:24,719 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:24,719 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:24,719 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:24,719 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:24,719 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:24,719 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:25,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:25,082 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:25,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:25,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50520 deadline: 1681844125083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:25,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:25,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844125083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:25,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:25,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844125083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:25,135 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304189a7f9a2af25b47f89d9904c44c419e12 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304189a7f9a2af25b47f89d9904c44c419e12 2023-04-18 18:54:25,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:25,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50520 deadline: 1681844125187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:25,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:25,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844125188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:25,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:25,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50520 deadline: 1681844125390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:25,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:25,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844125392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:25,541 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1157, memsize=58.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/83544b2f66d24fc88885a7c0f0f18514 2023-04-18 18:54:25,554 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=1157 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/3ebf71667e5c4e6bb9d1e6e746c75e07 2023-04-18 18:54:25,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:25,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50520 deadline: 1681844125695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:25,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:25,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844125696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:25,973 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=1157 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/c6f2cc26137e4aec8e78a6b007a1ae6c 2023-04-18 18:54:25,978 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/83544b2f66d24fc88885a7c0f0f18514 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/83544b2f66d24fc88885a7c0f0f18514 2023-04-18 18:54:25,983 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/83544b2f66d24fc88885a7c0f0f18514, entries=150, sequenceid=1157, filesize=25.4 K 2023-04-18 18:54:25,984 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/3ebf71667e5c4e6bb9d1e6e746c75e07 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/3ebf71667e5c4e6bb9d1e6e746c75e07 2023-04-18 18:54:25,988 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/3ebf71667e5c4e6bb9d1e6e746c75e07, entries=150, sequenceid=1157, filesize=11.9 K 2023-04-18 18:54:25,989 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/c6f2cc26137e4aec8e78a6b007a1ae6c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/c6f2cc26137e4aec8e78a6b007a1ae6c 2023-04-18 18:54:25,993 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/c6f2cc26137e4aec8e78a6b007a1ae6c, entries=150, sequenceid=1157, filesize=11.9 K 2023-04-18 18:54:25,994 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~174.43 KB/178620, heapSize ~477.28 KB/488736, currentSize=13.42 KB/13740 for cbf5abd278bb2be3a248e6824371aa01 in 1275ms, sequenceid=1157, compaction requested=true 2023-04-18 18:54:25,994 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:25,994 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:25,994 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:25,994 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:25,994 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:25,996 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:25,996 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:26,006 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,006 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,006 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:26,016 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:26,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4afd228e[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:26,018 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:26,018 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,019 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,019 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:26,019 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:26,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:26,020 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,020 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:26,021 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,021 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:26,021 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,021 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:26,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:26,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:26,025 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:26,025 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:26,025 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:26,025 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:26,025 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:26,029 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,029 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,030 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:26,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,030 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,032 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,036 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:26,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:26,036 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:26,036 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:26,036 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,036 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:26,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:26,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:26,036 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:26,036 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:26,036 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,036 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,036 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,036 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:26,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:26,132 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 18:54:26,133 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:26,133 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:26,133 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:26,133 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:26,133 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:26,133 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:26,136 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:26,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:26,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:26,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:26,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:26,152 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:26,152 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:26,152 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:26,152 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:26,153 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,153 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,154 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:26,154 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:26,154 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,154 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:26,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:26,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:26,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:26,160 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:26,160 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:26,160 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:26,164 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:26,164 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:26,164 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:26,164 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:26,164 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:26,164 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:26,164 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:26,164 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:26,164 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,164 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:26,164 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:26,166 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,166 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,166 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,166 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:26,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:26,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56a65fdf[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:26,168 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:26,168 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,169 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,169 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:26,169 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:26,169 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:26,169 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,169 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:26,171 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,171 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:26,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,171 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:26,171 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:26,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:26,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:26,173 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:26,173 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:26,173 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:26,173 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:26,173 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:26,174 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,174 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,175 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,175 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,176 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,178 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,178 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:26,178 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:26,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:26,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:26,178 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:26,178 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:26,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:26,178 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,179 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,179 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,179 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:26,179 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,179 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:26,179 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:26,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:26,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:26,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681844066159, max timestamp=1681844066202Num uniques 859; 2023-04-18 18:54:26,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844126201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681844066159, max timestamp=1681844066202Num uniques 859; 2023-04-18 18:54:26,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 18:54:26,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681844066159, max timestamp=1681844066202Num uniques 859; 2023-04-18 18:54:26,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 18:54:26,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 18:54:26,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:26,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844126219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:26,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 472 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844126223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:26,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50520 deadline: 1681844126223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:26,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844126224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,279 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:26,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:26,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:26,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:26,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:26,283 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:26,283 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:26,283 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:26,283 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:26,283 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,283 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,285 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:26,285 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:26,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:26,285 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,285 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:26,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:26,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:26,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:26,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:26,286 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:26,286 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:26,287 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:26,287 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:26,287 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:26,287 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:26,287 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:26,287 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:26,287 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:26,287 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:26,287 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:26,287 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,287 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:26,287 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:26,290 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,290 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,291 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,291 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:26,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:26,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fb37b72[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:26,292 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:26,292 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,293 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,293 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,293 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:26,293 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:26,293 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:26,293 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:26,295 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,295 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:26,295 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:26,295 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:26,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:26,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:26,297 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:26,298 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:26,298 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:26,298 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:26,298 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:26,299 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,299 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,299 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:26,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:26,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:26,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,300 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,302 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,305 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,305 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,305 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,305 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,305 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,305 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:26,305 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:26,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:26,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:26,305 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,306 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:26,306 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:26,306 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:26,305 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:26,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:26,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:26,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:26,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:26,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844126305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:26,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844126325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:26,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50520 deadline: 1681844126328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,331 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:26,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 474 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844126328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:26,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844126329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,406 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:26,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:26,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:26,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:26,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:26,410 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:26,410 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:26,410 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:26,410 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:26,411 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,411 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,413 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:26,413 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:26,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:26,413 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,413 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:26,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:26,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:26,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:26,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:26,414 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:26,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:26,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:26,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:26,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:26,415 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:26,415 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:26,415 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:26,415 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:26,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:26,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:26,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:26,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:26,418 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,418 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,419 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,419 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:26,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:26,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c94577e[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:26,421 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:26,421 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,422 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,423 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:26,423 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:26,423 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:26,423 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,423 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:26,430 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:26,430 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,430 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,430 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:26,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:26,433 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:26,433 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:26,433 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:26,433 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:26,433 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:26,435 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,435 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,435 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:26,436 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,437 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,440 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:26,440 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:26,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:26,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:26,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:26,440 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,440 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:26,440 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:26,440 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:26,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:26,440 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,440 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:26,440 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,440 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,441 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,441 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:26,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844126510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:26,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844126529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:26,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50520 deadline: 1681844126531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:26,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 476 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844126532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:26,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844126534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,541 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:26,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:26,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:26,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:26,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:26,544 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:26,544 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:26,544 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:26,544 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:26,545 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,545 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,546 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,546 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:26,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:26,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:26,546 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,546 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:26,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:26,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:26,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:26,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:26,548 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:26,548 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:26,548 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:26,548 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:26,548 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:26,548 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:26,548 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:26,548 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:26,548 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:26,548 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:26,548 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:26,549 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,549 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:26,553 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:26,556 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,556 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,557 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,557 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:26,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:26,558 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:26,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fdefd30[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:26,558 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,560 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,560 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:26,560 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:26,560 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:26,560 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,560 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:26,562 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:26,562 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:26,562 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,562 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:26,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,567 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418369b7fc552ea46b98da3563cafaa98d2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418369b7fc552ea46b98da3563cafaa98d2 2023-04-18 18:54:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:26,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:26,569 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:26,569 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:26,570 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:26,570 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:26,570 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:26,572 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,572 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:26,572 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,574 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1168, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/87e28cf4e53042f0ae2813e708e168ed 2023-04-18 18:54:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,574 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,576 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,580 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:26,580 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:26,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:26,580 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:26,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:26,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:26,581 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:26,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:26,581 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,581 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,581 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,581 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,581 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,581 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,581 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:26,581 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:26,681 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:26,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:26,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:26,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:26,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:26,685 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:26,685 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:26,685 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:26,685 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:26,686 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,686 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,687 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,687 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:26,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:26,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:26,687 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,687 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:26,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:26,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:26,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:26,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:26,689 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:26,689 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:26,689 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:26,689 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:26,689 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:26,689 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:26,689 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:26,689 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:26,689 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:26,689 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:26,689 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:26,689 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,689 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:26,689 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:26,691 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,691 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,692 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,692 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:26,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:26,693 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:26,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4be207b3[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:26,693 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,694 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,695 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:26,695 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:26,695 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:26,695 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,695 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:26,697 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,697 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:26,697 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:26,697 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:26,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:26,700 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:26,700 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:26,700 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:26,700 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:26,700 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:26,701 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,701 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,701 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:26,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:26,705 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,706 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,708 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,709 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,709 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,709 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:26,709 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:26,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:26,709 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,709 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,709 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,709 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,709 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:26,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:26,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:26,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:26,709 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:26,709 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:26,810 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:26,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:26,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:26,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:26,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:26,813 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:26,813 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:26,813 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:26,813 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:26,814 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,814 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,815 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:26,815 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:26,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:26,815 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,815 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:26,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:26,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:26,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:26,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:26,816 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:26,816 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:26,816 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:26,816 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:26,816 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:26,817 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:26,817 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:26,817 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:26,817 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:26,817 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:26,817 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:26,817 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,817 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:26,817 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:26,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,819 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:26,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f5e8d56[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:26,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:26,821 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:26,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844126814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,821 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,822 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,822 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:26,822 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,823 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:26,822 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:26,823 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:26,824 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:26,824 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,824 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:26,824 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:26,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:26,827 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:26,827 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:26,827 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:26,827 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:26,827 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:26,832 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,832 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,832 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:26,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:26,833 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,834 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:26,838 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:26,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:26,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:26,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:26,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:26,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,838 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:26,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:26,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:26,838 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:26,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:26,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:26,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:26,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844126837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:26,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50520 deadline: 1681844126837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:26,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844126837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:26,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 478 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844126839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,938 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:26,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:26,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:26,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:26,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:26,942 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:26,942 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:26,942 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:26,942 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:26,942 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,942 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,944 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:26,944 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:26,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:26,944 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,944 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:26,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:26,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:26,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:26,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:26,945 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:26,945 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:26,946 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:26,946 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:26,947 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,947 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,947 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,948 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:26,949 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:26,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@315f1664[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:26,949 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,951 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,951 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:26,951 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:26,951 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:26,952 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,952 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:26,953 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:26,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:26,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:26,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:26,955 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:26,956 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:26,956 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:26,956 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:26,956 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:26,957 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,957 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,957 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:26,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:26,958 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,959 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:26,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:26,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,961 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,961 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:26,961 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:26,962 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:26,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:26,962 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:26,962 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:26,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:26,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:26,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:26,962 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:26,962 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,962 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:26,962 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:26,962 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:26,962 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:26,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:26,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:26,989 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1168 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/b0d0eff462374d3bb84827d45b6c6575 2023-04-18 18:54:27,000 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1168 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/9ea95ecf15574cedb4655a2009f18ab9 2023-04-18 18:54:27,005 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/87e28cf4e53042f0ae2813e708e168ed as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/87e28cf4e53042f0ae2813e708e168ed 2023-04-18 18:54:27,010 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/87e28cf4e53042f0ae2813e708e168ed, entries=300, sequenceid=1168, filesize=46.1 K 2023-04-18 18:54:27,010 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-18 18:54:27,012 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/b0d0eff462374d3bb84827d45b6c6575 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/b0d0eff462374d3bb84827d45b6c6575 2023-04-18 18:54:27,016 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/b0d0eff462374d3bb84827d45b6c6575, entries=150, sequenceid=1168, filesize=11.9 K 2023-04-18 18:54:27,017 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/9ea95ecf15574cedb4655a2009f18ab9 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/9ea95ecf15574cedb4655a2009f18ab9 2023-04-18 18:54:27,021 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/9ea95ecf15574cedb4655a2009f18ab9, entries=150, sequenceid=1168, filesize=11.9 K 2023-04-18 18:54:27,022 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for cbf5abd278bb2be3a248e6824371aa01 in 890ms, sequenceid=1168, compaction requested=true 2023-04-18 18:54:27,038 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:27,038 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-18 18:54:27,038 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:27,038 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:27,038 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-18 18:54:27,038 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:27,041 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 208656 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-18 18:54:27,041 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): cbf5abd278bb2be3a248e6824371aa01/A is initiating minor compaction (all files) 2023-04-18 18:54:27,042 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cbf5abd278bb2be3a248e6824371aa01/A in TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:27,042 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/3357de25dcc2400aab3824afee4f5a6b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/fb95d579ce9f4593be4ba7426e8d89de, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/e7a20fd9aea344ac8513df25797352a9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/c81ada1ff9e64090906f93ce117e0f72, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/ca26529115ea4844ac4099537708e73d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/076d998a3f474df6967624d83c17e1a2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/83544b2f66d24fc88885a7c0f0f18514, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/87e28cf4e53042f0ae2813e708e168ed] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp, totalSize=203.8 K 2023-04-18 18:54:27,042 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 90464 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-18 18:54:27,042 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): cbf5abd278bb2be3a248e6824371aa01/B is initiating minor compaction (all files) 2023-04-18 18:54:27,042 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cbf5abd278bb2be3a248e6824371aa01/B in TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:27,042 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/2dbc95602be54157a186710e92fbb49c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/ee2b2ffdfa4144a995756dd2f012765b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/67be30e6125c4106be54fb0691433b7a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/0021b1b233394ded90cdf8792287c6f3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/62d845575eb548659248f3c82884665c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/bc42c0e5139e4d49901d4f27345f0341, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/3ebf71667e5c4e6bb9d1e6e746c75e07, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/b0d0eff462374d3bb84827d45b6c6575] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp, totalSize=88.3 K 2023-04-18 18:54:27,042 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 3357de25dcc2400aab3824afee4f5a6b, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1681844057530 2023-04-18 18:54:27,043 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting fb95d579ce9f4593be4ba7426e8d89de, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681844058212 2023-04-18 18:54:27,044 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 2dbc95602be54157a186710e92fbb49c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1681844057530 2023-04-18 18:54:27,044 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting e7a20fd9aea344ac8513df25797352a9, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1681844058840 2023-04-18 18:54:27,044 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting ee2b2ffdfa4144a995756dd2f012765b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681844058212 2023-04-18 18:54:27,044 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting c81ada1ff9e64090906f93ce117e0f72, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1681844060375 2023-04-18 18:54:27,044 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 67be30e6125c4106be54fb0691433b7a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1681844058840 2023-04-18 18:54:27,045 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting ca26529115ea4844ac4099537708e73d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=943, earliestPutTs=1681844061364 2023-04-18 18:54:27,045 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 0021b1b233394ded90cdf8792287c6f3, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1681844060375 2023-04-18 18:54:27,045 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 076d998a3f474df6967624d83c17e1a2, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=948, earliestPutTs=1681844061986 2023-04-18 18:54:27,045 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 62d845575eb548659248f3c82884665c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=943, earliestPutTs=1681844061364 2023-04-18 18:54:27,045 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 83544b2f66d24fc88885a7c0f0f18514, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1157, earliestPutTs=1681844063955 2023-04-18 18:54:27,045 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting bc42c0e5139e4d49901d4f27345f0341, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=948, earliestPutTs=1681844061986 2023-04-18 18:54:27,046 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 87e28cf4e53042f0ae2813e708e168ed, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=1168, earliestPutTs=1681844066094 2023-04-18 18:54:27,046 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 3ebf71667e5c4e6bb9d1e6e746c75e07, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1157, earliestPutTs=1681844063955 2023-04-18 18:54:27,046 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b0d0eff462374d3bb84827d45b6c6575, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1168, earliestPutTs=1681844066094 2023-04-18 18:54:27,062 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:27,063 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): cbf5abd278bb2be3a248e6824371aa01#B#compaction#165 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:27,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:27,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:27,076 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): cbf5abd278bb2be3a248e6824371aa01#A#compaction#164 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:27,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:27,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:27,077 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:27,077 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:27,077 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:27,077 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:27,078 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:27,078 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:27,081 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:27,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:27,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:27,081 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:27,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:27,082 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:27,082 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:27,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:27,082 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:27,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:27,086 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:27,086 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:27,086 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:27,086 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:27,086 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:27,086 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:27,086 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:27,086 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-18 18:54:27,086 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:27,087 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:27,087 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:27,087 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:27,087 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:27,087 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:27,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:27,329 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:27,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:27,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844127348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:27,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:27,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 480 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844127349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:27,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:27,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844127349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:27,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:27,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50520 deadline: 1681844127350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:27,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:27,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844127351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:27,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:27,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844127451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:27,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:27,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844127455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:27,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:27,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50520 deadline: 1681844127455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:27,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:27,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844127455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:27,489 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/55644aba3c284cd1b9dda4432c689ec3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/55644aba3c284cd1b9dda4432c689ec3 2023-04-18 18:54:27,495 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in cbf5abd278bb2be3a248e6824371aa01/B of cbf5abd278bb2be3a248e6824371aa01 into 55644aba3c284cd1b9dda4432c689ec3(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:27,495 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:27,495 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., storeName=cbf5abd278bb2be3a248e6824371aa01/B, priority=8, startTime=1681844067038; duration=0sec 2023-04-18 18:54:27,495 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:27,495 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-18 18:54:27,498 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 90464 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-18 18:54:27,498 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): cbf5abd278bb2be3a248e6824371aa01/C is initiating minor compaction (all files) 2023-04-18 18:54:27,498 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cbf5abd278bb2be3a248e6824371aa01/C in TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:27,498 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/577d83a5163d489d960cc06957720521, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/7ad8c3b0369e43a290f387d76fd30498, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/276135bd3cf744f8966f7c2dccc1cd55, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/8d202aa3c1e04bb4a23709fbaf8b4184, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/65a2bb22879d4f20ac3ddfff949699c6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/97c3768a16a34c6dbeccfecd397bb7cd, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/c6f2cc26137e4aec8e78a6b007a1ae6c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/9ea95ecf15574cedb4655a2009f18ab9] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp, totalSize=88.3 K 2023-04-18 18:54:27,499 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 577d83a5163d489d960cc06957720521, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1681844057530 2023-04-18 18:54:27,499 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 7ad8c3b0369e43a290f387d76fd30498, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681844058212 2023-04-18 18:54:27,499 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 276135bd3cf744f8966f7c2dccc1cd55, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1681844058840 2023-04-18 18:54:27,500 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 8d202aa3c1e04bb4a23709fbaf8b4184, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1681844060375 2023-04-18 18:54:27,500 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 65a2bb22879d4f20ac3ddfff949699c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=943, earliestPutTs=1681844061364 2023-04-18 18:54:27,501 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 97c3768a16a34c6dbeccfecd397bb7cd, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=948, earliestPutTs=1681844061986 2023-04-18 18:54:27,501 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting c6f2cc26137e4aec8e78a6b007a1ae6c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1157, earliestPutTs=1681844063955 2023-04-18 18:54:27,502 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 9ea95ecf15574cedb4655a2009f18ab9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1168, earliestPutTs=1681844066094 2023-04-18 18:54:27,515 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418ca019b76dd5e4e6ba31ce54fa0502744 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418ca019b76dd5e4e6ba31ce54fa0502744 2023-04-18 18:54:27,517 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): cbf5abd278bb2be3a248e6824371aa01#C#compaction#167 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:27,526 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/07c5b76b4cb84bd789128cde98708425 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/07c5b76b4cb84bd789128cde98708425 2023-04-18 18:54:27,530 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in cbf5abd278bb2be3a248e6824371aa01/C of cbf5abd278bb2be3a248e6824371aa01 into 07c5b76b4cb84bd789128cde98708425(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:27,531 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:27,531 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., storeName=cbf5abd278bb2be3a248e6824371aa01/C, priority=8, startTime=1681844067038; duration=0sec 2023-04-18 18:54:27,531 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:27,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:27,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844127656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:27,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:27,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50520 deadline: 1681844127658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:27,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:27,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844127659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:27,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:27,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844127659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:27,921 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1202, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/c5aca545f704448abce5ab8b6ae3431c 2023-04-18 18:54:27,932 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1202 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/ed69ec5f64d34f688aa0ab206c66a485 2023-04-18 18:54:27,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:27,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844127961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:27,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:27,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50520 deadline: 1681844127961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:27,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:27,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844127961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:27,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:27,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844127961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,309 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/cf41c80de3bf46cbbd5d17c44b710551 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/cf41c80de3bf46cbbd5d17c44b710551 2023-04-18 18:54:28,314 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in cbf5abd278bb2be3a248e6824371aa01/A of cbf5abd278bb2be3a248e6824371aa01 into cf41c80de3bf46cbbd5d17c44b710551(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-18 18:54:28,314 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:28,314 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., storeName=cbf5abd278bb2be3a248e6824371aa01/A, priority=8, startTime=1681844067038; duration=1sec 2023-04-18 18:54:28,314 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:28,344 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1202 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/9083fe56abf04dca9b9817aeaca9fe8a 2023-04-18 18:54:28,349 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/c5aca545f704448abce5ab8b6ae3431c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/c5aca545f704448abce5ab8b6ae3431c 2023-04-18 18:54:28,353 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/c5aca545f704448abce5ab8b6ae3431c, entries=150, sequenceid=1202, filesize=25.4 K 2023-04-18 18:54:28,354 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/ed69ec5f64d34f688aa0ab206c66a485 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/ed69ec5f64d34f688aa0ab206c66a485 2023-04-18 18:54:28,358 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/ed69ec5f64d34f688aa0ab206c66a485, entries=150, sequenceid=1202, filesize=11.9 K 2023-04-18 18:54:28,358 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/9083fe56abf04dca9b9817aeaca9fe8a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/9083fe56abf04dca9b9817aeaca9fe8a 2023-04-18 18:54:28,362 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/9083fe56abf04dca9b9817aeaca9fe8a, entries=150, sequenceid=1202, filesize=11.9 K 2023-04-18 18:54:28,363 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for cbf5abd278bb2be3a248e6824371aa01 in 1277ms, sequenceid=1202, compaction requested=false 2023-04-18 18:54:28,363 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:28,363 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:28,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:28,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:28,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:28,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:28,365 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,365 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:28,366 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:28,366 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:28,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,366 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-18 18:54:28,366 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,367 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:28,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:28,367 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:28,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:28,367 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:28,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:28,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:28,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69b04a02[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:28,367 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:28,367 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,369 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,369 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:28,370 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:28,370 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:28,370 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,370 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:28,371 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:28,371 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:28,371 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,371 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:28,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:28,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:28,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:28,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:28,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:28,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:28,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:28,377 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:28,377 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:28,378 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:28,378 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:28,378 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:28,380 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041811935a4d29e04ceb8c1ac1f7454bff5e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041811935a4d29e04ceb8c1ac1f7454bff5e 2023-04-18 18:54:28,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:28,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:28,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:28,381 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:28,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:28,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:28,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:28,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,383 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,386 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,386 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,386 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,386 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,386 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,386 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,386 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,386 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:28,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:28,387 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:28,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:28,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:28,387 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:28,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:28,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:28,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:28,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:28,387 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:28,387 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:28,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:28,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 500 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844128411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:28,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844128464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:28,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844128465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:28,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844128466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:28,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50520 deadline: 1681844128467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,487 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:28,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:28,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:28,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:28,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:28,491 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:28,491 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:28,491 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:28,491 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:28,491 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,491 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,493 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,493 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:28,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:28,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:28,493 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,493 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:28,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:28,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:28,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:28,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:28,495 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:28,495 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:28,495 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:28,495 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:28,495 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:28,495 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:28,495 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:28,495 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:28,495 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:28,495 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:28,495 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:28,495 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,495 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:28,495 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:28,497 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,497 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:28,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:28,498 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,498 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:28,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:28,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:28,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:28,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:28,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32340718[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:28,499 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:28,499 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,502 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,502 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:28,502 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:28,502 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,502 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:28,502 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:28,504 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,504 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:28,504 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:28,504 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:28,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:28,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:28,508 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:28,508 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:28,508 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:28,508 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:28,508 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:28,510 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,510 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:28,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:28,510 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:28,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:28,511 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:28,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,512 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:28,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:28,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 502 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844128515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,518 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:28,518 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:28,518 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:28,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:28,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:28,518 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:28,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:28,518 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,519 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,519 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,519 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:28,519 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:28,519 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,519 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,519 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:28,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:28,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:28,619 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:28,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:28,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:28,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:28,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:28,622 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:28,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:28,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:28,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:28,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,623 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,624 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,624 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:28,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:28,624 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,624 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:28,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:28,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:28,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:28,625 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:28,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:28,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:28,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:28,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:28,626 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:28,626 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:28,626 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:28,626 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:28,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:28,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:28,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:28,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:28,627 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,627 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:28,628 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,628 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:28,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:28,629 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:28,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@596b0835[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:28,629 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,630 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,631 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:28,631 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:28,631 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:28,631 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,631 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:28,633 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,633 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,633 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:28,633 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:28,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:28,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:28,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:28,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:28,635 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:28,635 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:28,635 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:28,635 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:28,635 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:28,637 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,637 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:28,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:28,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:28,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,644 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:28,644 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:28,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,646 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,648 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,648 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,648 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:28,648 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:28,648 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,648 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,648 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,648 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,648 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:28,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:28,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:28,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:28,648 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:28,649 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:28,649 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:28,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:28,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:28,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 504 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844128720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,749 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:28,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:28,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:28,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:28,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:28,752 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:28,752 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:28,752 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:28,752 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:28,753 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,753 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,754 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:28,754 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:28,754 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,754 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:28,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:28,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:28,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:28,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:28,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:28,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:28,762 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:28,763 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:28,763 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:28,763 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:28,763 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:28,763 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:28,763 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:28,763 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:28,763 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,763 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:28,763 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:28,766 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,766 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:28,766 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,766 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:28,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:28,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ced94a[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:28,768 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:28,768 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,769 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,769 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:28,769 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:28,769 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:28,770 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,770 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:28,771 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,771 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:28,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:28,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:28,771 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:28,771 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:28,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:28,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:28,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:28,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:28,774 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:28,774 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:28,774 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:28,774 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:28,774 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:28,775 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,775 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:28,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:28,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:28,776 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,778 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,782 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:28,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:28,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:28,782 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,782 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,782 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,782 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,782 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,782 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,782 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:28,782 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:28,782 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:28,782 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:28,782 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:28,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:28,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:28,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:28,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:28,788 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1217, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/d0762f8d0484476498aa0bf541ea37f1 2023-04-18 18:54:28,882 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:28,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:28,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:28,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:28,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:28,886 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:28,886 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:28,886 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:28,886 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:28,887 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,887 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,888 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:28,888 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:28,888 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,888 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:28,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:28,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:28,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:28,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:28,890 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:28,890 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:28,890 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:28,890 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:28,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:28,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:28,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:28,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:28,891 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,891 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:28,892 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,892 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:28,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:28,893 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:28,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bc5610b[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:28,893 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,894 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,895 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:28,895 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:28,895 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:28,895 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,895 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:28,897 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,897 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:28,897 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:28,897 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:28,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:28,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:28,900 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:28,900 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:28,900 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:28,900 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:28,900 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:28,901 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,901 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:28,901 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,902 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:28,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:28,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,903 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:28,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:28,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,908 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,908 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,908 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:28,908 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:28,908 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:28,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:28,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:28,908 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:28,908 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:28,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:28,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:28,908 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:28,908 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:28,908 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:28,909 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:28,908 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:28,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:28,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:29,009 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:29,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:29,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:29,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:29,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:29,012 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:29,012 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:29,012 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:29,012 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:29,013 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,013 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,018 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:29,018 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:29,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:29,018 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,018 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:29,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:29,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:29,019 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:29,020 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:29,020 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:29,020 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:29,020 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:29,022 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,022 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:29,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:29,022 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,022 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:29,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:29,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e12b2df[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:29,024 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:29,024 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,025 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:29,026 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,026 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:29,027 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,027 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:29,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:29,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:29,027 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:29,027 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:29,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:29,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:29,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:29,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 506 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844129025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,030 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:29,030 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:29,030 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:29,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:29,030 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:29,030 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:29,032 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,032 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:29,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,032 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:29,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:29,033 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,034 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:29,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:29,037 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:29,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:29,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:29,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:29,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:29,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,037 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:29,037 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:29,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:29,137 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:29,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:29,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:29,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:29,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:29,141 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:29,141 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:29,141 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:29,141 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:29,141 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,141 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,142 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,142 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:29,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:29,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:29,143 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,143 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:29,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:29,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:29,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:29,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:29,144 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:29,144 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:29,144 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:29,144 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:29,144 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:29,144 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:29,144 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:29,144 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:29,144 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:29,144 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:29,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:29,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:29,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:29,146 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,146 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:29,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:29,146 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,146 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:29,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:29,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:29,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:29,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:29,148 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:29,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e0f472f[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:29,148 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,149 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,149 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:29,149 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:29,149 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:29,149 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,149 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:29,151 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,151 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:29,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:29,151 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:29,152 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:29,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:29,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:29,154 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:29,154 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:29,154 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:29,154 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:29,154 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:29,156 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,156 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:29,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:29,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,156 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:29,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:29,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:29,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:29,157 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,165 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,167 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:29,167 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:29,167 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:29,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:29,168 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:29,168 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:29,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:29,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:29,168 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,168 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,168 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,168 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,168 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,168 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:29,168 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:29,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:29,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:29,200 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1217 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/b141b33cb4eb4621b8ad5c9ef0669c61 2023-04-18 18:54:29,268 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:29,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:29,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:29,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:29,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:29,271 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:29,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:29,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:29,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:29,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,274 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:29,274 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:29,274 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,274 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:29,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:29,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:29,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:29,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:29,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:29,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:29,276 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:29,276 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:29,276 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:29,276 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:29,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:29,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:29,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:29,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:29,282 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,282 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:29,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:29,283 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,283 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:29,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:29,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:29,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:29,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:29,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a4d7f97[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:29,287 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:29,287 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,289 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,289 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:29,289 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:29,289 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:29,289 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,289 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:29,291 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:29,291 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:29,291 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,291 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:29,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:29,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:29,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:29,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:29,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:29,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:29,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:29,295 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:29,295 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:29,295 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:29,295 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:29,295 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:29,300 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,300 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:29,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:29,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,300 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:29,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:29,301 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:29,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:29,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,302 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,305 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,305 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:29,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:29,305 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:29,305 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,305 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:29,306 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,306 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:29,306 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:29,306 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,306 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:29,306 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,306 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:29,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:29,406 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:29,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:29,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:29,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:29,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:29,410 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:29,410 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:29,410 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:29,410 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:29,411 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,411 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,412 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,412 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:29,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:29,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:29,412 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,412 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:29,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:29,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:29,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:29,413 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:29,413 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:29,413 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:29,414 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:29,414 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:29,414 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:29,414 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:29,414 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:29,414 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:29,414 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:29,414 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:29,414 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,414 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:29,414 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:29,418 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,418 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:29,419 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,419 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:29,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:29,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:29,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:29,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:29,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@344306e[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:29,420 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:29,420 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,421 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,422 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:29,422 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,422 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:29,422 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:29,422 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:29,423 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:29,423 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:29,423 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:29,423 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:29,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:29,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:29,426 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:29,426 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:29,426 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:29,426 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:29,426 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:29,429 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,429 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,429 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:29,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:29,429 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:29,430 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:29,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,433 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,433 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:29,433 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,433 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,433 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,433 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,433 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:29,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:29,433 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:29,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:29,433 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,433 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:29,433 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:29,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:29,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:29,433 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:29,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:29,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844129468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:29,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844129469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:29,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844129475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:29,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50520 deadline: 1681844129478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,533 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:29,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:29,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:29,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:29,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 508 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844129534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:29,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:29,536 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:29,536 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:29,536 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:29,537 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:29,537 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,537 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,538 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,538 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:29,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:29,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:29,539 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,539 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:29,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:29,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:29,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:29,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:29,540 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:29,540 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:29,540 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:29,540 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:29,540 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:29,540 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:29,540 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:29,540 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:29,540 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:29,540 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:29,540 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:29,540 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,540 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:29,540 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:29,542 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,542 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:29,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:29,542 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:29,542 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:29,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:29,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:29,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:29,544 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:29,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@185bee93[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:29,544 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,545 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,546 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:29,546 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:29,546 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,546 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:29,546 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:29,547 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,547 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:29,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:29,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:29,547 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:29,547 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:29,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:29,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:29,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:29,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:29,549 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:29,549 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:29,549 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:29,549 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:29,549 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:29,551 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,551 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,551 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:29,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:29,553 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,554 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,560 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:29,560 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,560 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:29,560 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:29,560 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:29,561 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:29,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:29,561 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:29,561 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,561 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:29,561 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,561 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:29,561 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:29,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:29,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:29,611 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1217 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/b0c245d0ac3c43f687ac257a3b6416a6 2023-04-18 18:54:29,616 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/d0762f8d0484476498aa0bf541ea37f1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/d0762f8d0484476498aa0bf541ea37f1 2023-04-18 18:54:29,621 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/d0762f8d0484476498aa0bf541ea37f1, entries=200, sequenceid=1217, filesize=32.3 K 2023-04-18 18:54:29,622 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/b141b33cb4eb4621b8ad5c9ef0669c61 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/b141b33cb4eb4621b8ad5c9ef0669c61 2023-04-18 18:54:29,630 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/b141b33cb4eb4621b8ad5c9ef0669c61, entries=150, sequenceid=1217, filesize=11.9 K 2023-04-18 18:54:29,631 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/b0c245d0ac3c43f687ac257a3b6416a6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/b0c245d0ac3c43f687ac257a3b6416a6 2023-04-18 18:54:29,638 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/b0c245d0ac3c43f687ac257a3b6416a6, entries=150, sequenceid=1217, filesize=11.9 K 2023-04-18 18:54:29,641 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for cbf5abd278bb2be3a248e6824371aa01 in 1275ms, sequenceid=1217, compaction requested=true 2023-04-18 18:54:29,641 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:29,641 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:29,641 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:54:29,641 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:29,641 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:54:29,641 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:29,642 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85758 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:54:29,642 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): cbf5abd278bb2be3a248e6824371aa01/A is initiating minor compaction (all files) 2023-04-18 18:54:29,642 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cbf5abd278bb2be3a248e6824371aa01/A in TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:29,643 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/cf41c80de3bf46cbbd5d17c44b710551, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/c5aca545f704448abce5ab8b6ae3431c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/d0762f8d0484476498aa0bf541ea37f1] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp, totalSize=83.7 K 2023-04-18 18:54:29,643 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting cf41c80de3bf46cbbd5d17c44b710551, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=1168, earliestPutTs=1681844066094 2023-04-18 18:54:29,644 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:54:29,644 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): cbf5abd278bb2be3a248e6824371aa01/B is initiating minor compaction (all files) 2023-04-18 18:54:29,644 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cbf5abd278bb2be3a248e6824371aa01/B in TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:29,644 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/55644aba3c284cd1b9dda4432c689ec3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/ed69ec5f64d34f688aa0ab206c66a485, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/b141b33cb4eb4621b8ad5c9ef0669c61] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp, totalSize=36.3 K 2023-04-18 18:54:29,645 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 55644aba3c284cd1b9dda4432c689ec3, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1168, earliestPutTs=1681844066094 2023-04-18 18:54:29,645 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting c5aca545f704448abce5ab8b6ae3431c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1202, earliestPutTs=1681844066216 2023-04-18 18:54:29,645 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting ed69ec5f64d34f688aa0ab206c66a485, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1202, earliestPutTs=1681844066216 2023-04-18 18:54:29,646 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting d0762f8d0484476498aa0bf541ea37f1, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1217, earliestPutTs=1681844067348 2023-04-18 18:54:29,646 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b141b33cb4eb4621b8ad5c9ef0669c61, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1217, earliestPutTs=1681844067348 2023-04-18 18:54:29,661 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:29,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:29,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:29,665 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): cbf5abd278bb2be3a248e6824371aa01#B#compaction#174 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:29,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:29,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:29,668 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:29,668 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:29,668 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:29,668 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:29,668 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,668 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,671 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:29,671 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,671 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:29,671 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:29,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:29,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:29,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:29,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:29,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:29,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:29,684 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:29,684 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:29,684 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:29,689 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): cbf5abd278bb2be3a248e6824371aa01#A#compaction#173 average throughput is 0.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:29,690 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:29,691 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:29,691 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:29,691 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:29,691 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 18:54:29,691 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:29,691 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:29,691 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:29,691 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:29,691 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:29,691 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:29,707 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/88cdefb4bae14576a29e4fd5420621fc as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/88cdefb4bae14576a29e4fd5420621fc 2023-04-18 18:54:29,714 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cbf5abd278bb2be3a248e6824371aa01/B of cbf5abd278bb2be3a248e6824371aa01 into 88cdefb4bae14576a29e4fd5420621fc(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:29,715 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:29,715 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., storeName=cbf5abd278bb2be3a248e6824371aa01/B, priority=13, startTime=1681844069641; duration=0sec 2023-04-18 18:54:29,715 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:29,715 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:54:29,716 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:54:29,716 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): cbf5abd278bb2be3a248e6824371aa01/C is initiating minor compaction (all files) 2023-04-18 18:54:29,716 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cbf5abd278bb2be3a248e6824371aa01/C in TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:29,717 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/07c5b76b4cb84bd789128cde98708425, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/9083fe56abf04dca9b9817aeaca9fe8a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/b0c245d0ac3c43f687ac257a3b6416a6] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp, totalSize=36.3 K 2023-04-18 18:54:29,717 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 07c5b76b4cb84bd789128cde98708425, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1168, earliestPutTs=1681844066094 2023-04-18 18:54:29,717 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 9083fe56abf04dca9b9817aeaca9fe8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1202, earliestPutTs=1681844066216 2023-04-18 18:54:29,718 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b0c245d0ac3c43f687ac257a3b6416a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1217, earliestPutTs=1681844067348 2023-04-18 18:54:29,740 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): cbf5abd278bb2be3a248e6824371aa01#C#compaction#176 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:29,756 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/795fd43d95ca47609a61262b9cc4132b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/795fd43d95ca47609a61262b9cc4132b 2023-04-18 18:54:29,762 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cbf5abd278bb2be3a248e6824371aa01/C of cbf5abd278bb2be3a248e6824371aa01 into 795fd43d95ca47609a61262b9cc4132b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:29,762 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:29,762 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., storeName=cbf5abd278bb2be3a248e6824371aa01/C, priority=13, startTime=1681844069641; duration=0sec 2023-04-18 18:54:29,763 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:30,150 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418e60525ab5eb045b9a9d809dcb918a865 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e60525ab5eb045b9a9d809dcb918a865 2023-04-18 18:54:30,159 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/38f5b5b052ac4daa953956b4edb2ede3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/38f5b5b052ac4daa953956b4edb2ede3 2023-04-18 18:54:30,165 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cbf5abd278bb2be3a248e6824371aa01/A of cbf5abd278bb2be3a248e6824371aa01 into 38f5b5b052ac4daa953956b4edb2ede3(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:30,165 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:30,165 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., storeName=cbf5abd278bb2be3a248e6824371aa01/A, priority=13, startTime=1681844069641; duration=0sec 2023-04-18 18:54:30,165 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:30,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:30,541 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:30,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:30,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 519 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844130557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:30,559 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1237, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/7d51a1ef36214e7fb03ca734f85034a3 2023-04-18 18:54:30,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:30,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 521 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844130658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:30,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:30,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 523 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844130861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:30,970 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1237 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/8a00f6e2f02d442b8934460012ea6617 2023-04-18 18:54:30,982 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1237 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/3e91ed35ad77414a9b056c7ababf5053 2023-04-18 18:54:30,987 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/7d51a1ef36214e7fb03ca734f85034a3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/7d51a1ef36214e7fb03ca734f85034a3 2023-04-18 18:54:30,991 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/7d51a1ef36214e7fb03ca734f85034a3, entries=150, sequenceid=1237, filesize=25.4 K 2023-04-18 18:54:30,992 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/8a00f6e2f02d442b8934460012ea6617 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/8a00f6e2f02d442b8934460012ea6617 2023-04-18 18:54:30,997 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/8a00f6e2f02d442b8934460012ea6617, entries=150, sequenceid=1237, filesize=11.9 K 2023-04-18 18:54:30,998 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/3e91ed35ad77414a9b056c7ababf5053 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/3e91ed35ad77414a9b056c7ababf5053 2023-04-18 18:54:31,003 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/3e91ed35ad77414a9b056c7ababf5053, entries=150, sequenceid=1237, filesize=11.9 K 2023-04-18 18:54:31,004 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for cbf5abd278bb2be3a248e6824371aa01 in 1313ms, sequenceid=1237, compaction requested=false 2023-04-18 18:54:31,004 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:31,004 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:31,004 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:31,004 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:31,004 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,004 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:31,004 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:31,006 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,006 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:31,006 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:31,006 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:31,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:31,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:31,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:31,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:31,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cc9e488[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:31,008 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:31,008 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:31,009 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:31,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:31,009 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:31,009 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:31,009 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:31,010 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,010 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:31,011 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,011 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:31,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:31,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:31,011 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:31,011 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:31,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:31,014 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:31,014 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:31,014 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:31,014 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:31,014 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:31,017 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:31,017 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:31,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:31,017 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:31,018 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:31,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:31,019 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,021 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,021 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:31,021 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:31,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:31,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:31,021 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:31,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:31,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:31,021 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:31,022 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:31,022 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:31,021 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:31,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:31,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:31,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:31,022 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,022 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:31,022 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:31,022 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:31,122 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:31,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:31,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:31,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:31,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:31,125 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:31,125 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:31,125 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:31,125 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:31,126 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:31,126 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:31,127 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:31,127 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:31,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:31,128 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,128 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:31,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:31,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:31,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:31,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:31,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:31,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:31,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:31,129 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:31,129 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:31,129 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:31,129 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:31,129 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:31,129 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:31,129 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:31,129 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-18 18:54:31,129 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:31,129 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:31,129 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:31,129 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:31,129 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:31,130 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:31,144 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418728fd867af624b86bf3596757bb18bc3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418728fd867af624b86bf3596757bb18bc3 2023-04-18 18:54:31,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:31,167 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:31,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:31,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 542 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844131209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:31,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 544 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844131311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:31,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50484 deadline: 1681844131475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,479 DEBUG [Thread-851] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4128 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:54:31,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:31,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50448 deadline: 1681844131491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,493 DEBUG [Thread-855] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:54:31,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:31,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50514 deadline: 1681844131494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:31,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50520 deadline: 1681844131494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,498 DEBUG [Thread-853] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:54:31,499 DEBUG [Thread-847] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:54:31,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:31,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 546 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844131513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,549 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1252, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/c2ebc8825a0942e7950c18c6976c036f 2023-04-18 18:54:31,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:31,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 548 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844131818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:31,961 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1252 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/cd37f1578ba344438cf181a997ef769d 2023-04-18 18:54:31,979 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1252 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/76f616c7d80644259d11d230756fa213 2023-04-18 18:54:31,984 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/c2ebc8825a0942e7950c18c6976c036f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/c2ebc8825a0942e7950c18c6976c036f 2023-04-18 18:54:31,989 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/c2ebc8825a0942e7950c18c6976c036f, entries=150, sequenceid=1252, filesize=25.4 K 2023-04-18 18:54:31,990 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/cd37f1578ba344438cf181a997ef769d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/cd37f1578ba344438cf181a997ef769d 2023-04-18 18:54:31,994 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/cd37f1578ba344438cf181a997ef769d, entries=150, sequenceid=1252, filesize=11.9 K 2023-04-18 18:54:31,995 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/76f616c7d80644259d11d230756fa213 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/76f616c7d80644259d11d230756fa213 2023-04-18 18:54:32,011 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/76f616c7d80644259d11d230756fa213, entries=150, sequenceid=1252, filesize=11.9 K 2023-04-18 18:54:32,012 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for cbf5abd278bb2be3a248e6824371aa01 in 883ms, sequenceid=1252, compaction requested=true 2023-04-18 18:54:32,012 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:32,012 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:32,012 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:32,012 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:32,012 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:32,012 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:32,012 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:32,013 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:32,013 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:32,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:32,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:32,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:32,014 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:32,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:32,014 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:32,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:32,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:32,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:32,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:32,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:32,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d798541[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:32,015 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:32,015 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:32,019 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:32,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:32,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:32,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:32,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:32,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:32,020 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:32,020 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:32,021 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:32,021 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:32,021 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:32,021 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:32,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:32,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:32,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:32,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:32,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:32,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:32,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:32,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:32,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:32,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:32,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:32,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:32,024 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:32,024 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:32,024 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:32,024 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:32,024 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:32,027 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:32,027 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:32,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:32,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:32,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:32,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:32,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:32,027 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:32,028 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:32,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:32,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:32,029 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:32,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:32,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:32,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:32,031 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:32,031 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:32,031 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:32,031 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:32,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:32,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:32,031 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:32,031 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:32,031 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:32,031 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:32,031 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:32,031 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:32,032 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:32,031 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:32,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:32,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:32,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:32,132 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:32,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:32,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:32,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:32,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:32,135 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:32,135 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:32,135 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:32,135 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:32,136 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:32,136 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:32,137 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:32,137 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:32,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:32,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:32,138 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:32,138 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:32,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:32,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:32,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:32,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:32,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:32,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:32,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:32,139 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:32,139 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:32,139 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:32,139 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:32,139 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:32,139 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:32,140 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. started... 2023-04-18 18:54:32,140 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 18:54:32,140 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:32,140 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:32,140 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:32,140 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:32,140 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:32,140 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:32,157 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418a1cb61b378704beb97b7d3fcac90f54a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a1cb61b378704beb97b7d3fcac90f54a 2023-04-18 18:54:32,163 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1272, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/a1aaaf61d0b643ffb00f8eb9aa8b17d9 2023-04-18 18:54:32,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:32,327 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. as already flushing 2023-04-18 18:54:32,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:32,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 559 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844132347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:32,433 DEBUG [Thread-862] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x204b5880 to 127.0.0.1:65091 2023-04-18 18:54:32,433 DEBUG [Thread-862] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:54:32,434 DEBUG [Thread-860] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b416c44 to 127.0.0.1:65091 2023-04-18 18:54:32,434 DEBUG [Thread-860] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:54:32,435 DEBUG [Thread-858] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2668d41d to 127.0.0.1:65091 2023-04-18 18:54:32,435 DEBUG [Thread-858] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:54:32,438 DEBUG [Thread-864] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41173aec to 127.0.0.1:65091 2023-04-18 18:54:32,438 DEBUG [Thread-864] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:54:32,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:32,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 561 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844132449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:32,578 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1272 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/50f442b6204c41a19ff0955e8f4319a0 2023-04-18 18:54:32,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:32,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 563 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844132651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:32,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:32,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 565 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50444 deadline: 1681844132953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:32,989 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1272 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/28b1a650ccce4f369872f8f26bc4894a 2023-04-18 18:54:32,993 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/a1aaaf61d0b643ffb00f8eb9aa8b17d9 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/a1aaaf61d0b643ffb00f8eb9aa8b17d9 2023-04-18 18:54:32,996 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/a1aaaf61d0b643ffb00f8eb9aa8b17d9, entries=150, sequenceid=1272, filesize=25.4 K 2023-04-18 18:54:32,997 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/50f442b6204c41a19ff0955e8f4319a0 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/50f442b6204c41a19ff0955e8f4319a0 2023-04-18 18:54:33,000 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/50f442b6204c41a19ff0955e8f4319a0, entries=150, sequenceid=1272, filesize=11.9 K 2023-04-18 18:54:33,001 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/28b1a650ccce4f369872f8f26bc4894a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/28b1a650ccce4f369872f8f26bc4894a 2023-04-18 18:54:33,004 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/28b1a650ccce4f369872f8f26bc4894a, entries=150, sequenceid=1272, filesize=11.9 K 2023-04-18 18:54:33,005 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for cbf5abd278bb2be3a248e6824371aa01 in 865ms, sequenceid=1272, compaction requested=true 2023-04-18 18:54:33,005 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:33,005 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:33,005 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:33,005 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:33,005 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:33,005 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:33,005 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:33,006 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:33,006 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:33,007 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:33,007 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:33,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:33,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:33,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:33,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:33,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:33,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:33,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f1c15c0[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:33,008 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:33,008 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:33,009 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:33,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:33,009 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:33,009 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:33,009 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:33,009 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:33,009 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:33,010 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:33,010 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:33,010 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:33,010 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:33,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:33,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:33,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:33,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:33,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:33,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:33,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:33,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:33,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:33,013 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:33,013 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:33,013 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:33,013 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:33,013 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:33,015 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:33,015 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:33,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:33,015 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:33,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:33,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:33,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:33,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:33,015 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:33,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:33,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:33,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:33,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:33,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:33,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:33,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:33,016 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:33,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:33,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:33,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:33,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:33,018 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:33,018 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:33,018 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:33,018 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:33,018 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:33,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:33,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:33,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:33,018 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:33,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:33,018 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:33,019 DEBUG [Thread-857] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:33,019 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:33,019 DEBUG [Thread-857] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:33,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:33,019 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:33,019 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:33,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:33,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:33,119 DEBUG [Thread-857] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:33,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:33,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:33,456 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 18:54:33,456 DEBUG [Thread-849] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52d98fed to 127.0.0.1:65091 2023-04-18 18:54:33,456 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:33,456 DEBUG [Thread-849] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:54:33,456 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:33,456 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:33,456 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:33,456 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:33,456 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:33,868 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304188411bfa393cf4b3e84a0c13ce5491258 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304188411bfa393cf4b3e84a0c13ce5491258 2023-04-18 18:54:34,274 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1285, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/e2b999a59df84b6cb03b568598a63ae9 2023-04-18 18:54:34,683 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1285 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/3f0dd9c329dd4ecb94dfd74a202bfb08 2023-04-18 18:54:35,093 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1285 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/d582f578069b48a598a32779a3195eeb 2023-04-18 18:54:35,097 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/e2b999a59df84b6cb03b568598a63ae9 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/e2b999a59df84b6cb03b568598a63ae9 2023-04-18 18:54:35,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/e2b999a59df84b6cb03b568598a63ae9, entries=150, sequenceid=1285, filesize=25.4 K 2023-04-18 18:54:35,101 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/3f0dd9c329dd4ecb94dfd74a202bfb08 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/3f0dd9c329dd4ecb94dfd74a202bfb08 2023-04-18 18:54:35,105 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/3f0dd9c329dd4ecb94dfd74a202bfb08, entries=150, sequenceid=1285, filesize=11.9 K 2023-04-18 18:54:35,106 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/d582f578069b48a598a32779a3195eeb as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/d582f578069b48a598a32779a3195eeb 2023-04-18 18:54:35,110 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/d582f578069b48a598a32779a3195eeb, entries=150, sequenceid=1285, filesize=11.9 K 2023-04-18 18:54:35,111 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for cbf5abd278bb2be3a248e6824371aa01 in 1655ms, sequenceid=1285, compaction requested=true 2023-04-18 18:54:35,111 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:35,111 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:35,111 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:35,111 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:54:35,111 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:54:35,111 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:35,112 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 130912 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:54:35,112 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61612 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:54:35,112 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): cbf5abd278bb2be3a248e6824371aa01/A is initiating minor compaction (all files) 2023-04-18 18:54:35,112 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): cbf5abd278bb2be3a248e6824371aa01/B is initiating minor compaction (all files) 2023-04-18 18:54:35,113 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cbf5abd278bb2be3a248e6824371aa01/A in TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:35,113 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cbf5abd278bb2be3a248e6824371aa01/B in TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:35,113 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/38f5b5b052ac4daa953956b4edb2ede3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/7d51a1ef36214e7fb03ca734f85034a3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/c2ebc8825a0942e7950c18c6976c036f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/a1aaaf61d0b643ffb00f8eb9aa8b17d9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/e2b999a59df84b6cb03b568598a63ae9] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp, totalSize=127.8 K 2023-04-18 18:54:35,113 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/88cdefb4bae14576a29e4fd5420621fc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/8a00f6e2f02d442b8934460012ea6617, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/cd37f1578ba344438cf181a997ef769d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/50f442b6204c41a19ff0955e8f4319a0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/3f0dd9c329dd4ecb94dfd74a202bfb08] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp, totalSize=60.2 K 2023-04-18 18:54:35,113 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 38f5b5b052ac4daa953956b4edb2ede3, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=1217, earliestPutTs=1681844067348 2023-04-18 18:54:35,113 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 88cdefb4bae14576a29e4fd5420621fc, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=1217, earliestPutTs=1681844067348 2023-04-18 18:54:35,113 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 7d51a1ef36214e7fb03ca734f85034a3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1237, earliestPutTs=1681844068400 2023-04-18 18:54:35,113 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 8a00f6e2f02d442b8934460012ea6617, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1237, earliestPutTs=1681844068400 2023-04-18 18:54:35,113 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting c2ebc8825a0942e7950c18c6976c036f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1252, earliestPutTs=1681844070540 2023-04-18 18:54:35,114 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting cd37f1578ba344438cf181a997ef769d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1252, earliestPutTs=1681844070540 2023-04-18 18:54:35,114 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting a1aaaf61d0b643ffb00f8eb9aa8b17d9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1272, earliestPutTs=1681844071203 2023-04-18 18:54:35,114 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 50f442b6204c41a19ff0955e8f4319a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1272, earliestPutTs=1681844071203 2023-04-18 18:54:35,114 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting e2b999a59df84b6cb03b568598a63ae9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1285, earliestPutTs=1681844072339 2023-04-18 18:54:35,114 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 3f0dd9c329dd4ecb94dfd74a202bfb08, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1285, earliestPutTs=1681844072339 2023-04-18 18:54:35,126 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): cbf5abd278bb2be3a248e6824371aa01#B#compaction#189 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:35,127 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): cbf5abd278bb2be3a248e6824371aa01#A#compaction#188 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:35,488 DEBUG [Thread-851] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18496e17 to 127.0.0.1:65091 2023-04-18 18:54:35,488 DEBUG [Thread-851] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:54:35,519 DEBUG [Thread-855] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40e0a3dc to 127.0.0.1:65091 2023-04-18 18:54:35,520 DEBUG [Thread-855] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:54:35,531 DEBUG [Thread-847] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63ebf1e9 to 127.0.0.1:65091 2023-04-18 18:54:35,532 DEBUG [Thread-847] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:54:35,534 DEBUG [Thread-853] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25441895 to 127.0.0.1:65091 2023-04-18 18:54:35,534 DEBUG [Thread-853] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:54:35,534 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-18 18:54:35,535 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 110 2023-04-18 18:54:35,535 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 454 2023-04-18 18:54:35,535 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 201 2023-04-18 18:54:35,535 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 232 2023-04-18 18:54:35,535 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 213 2023-04-18 18:54:35,535 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-18 18:54:35,535 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(398): read 4543 2023-04-18 18:54:35,535 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(398): read 4723 2023-04-18 18:54:35,535 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-18 18:54:35,535 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(402): scanned 1961 2023-04-18 18:54:35,535 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(403): verified 5879 rows 2023-04-18 18:54:35,535 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(402): scanned 1986 2023-04-18 18:54:35,535 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(403): verified 5957 rows 2023-04-18 18:54:35,535 INFO [Listener at localhost/36951] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-18 18:54:35,535 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3ab850e7 to 127.0.0.1:65091 2023-04-18 18:54:35,535 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:54:35,536 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/7a009b6083ce4bfeb51572452325ed3a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/7a009b6083ce4bfeb51572452325ed3a 2023-04-18 18:54:35,539 INFO [Listener at localhost/36951] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-18 18:54:35,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-18 18:54:35,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-18 18:54:35,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-18 18:54:35,546 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844075546"}]},"ts":"1681844075546"} 2023-04-18 18:54:35,547 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in cbf5abd278bb2be3a248e6824371aa01/B of cbf5abd278bb2be3a248e6824371aa01 into 7a009b6083ce4bfeb51572452325ed3a(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:35,547 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:35,547 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., storeName=cbf5abd278bb2be3a248e6824371aa01/B, priority=11, startTime=1681844075111; duration=0sec 2023-04-18 18:54:35,547 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:35,547 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:54:35,548 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-18 18:54:35,550 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61612 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:54:35,550 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): cbf5abd278bb2be3a248e6824371aa01/C is initiating minor compaction (all files) 2023-04-18 18:54:35,550 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cbf5abd278bb2be3a248e6824371aa01/C in TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:35,550 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/795fd43d95ca47609a61262b9cc4132b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/3e91ed35ad77414a9b056c7ababf5053, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/76f616c7d80644259d11d230756fa213, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/28b1a650ccce4f369872f8f26bc4894a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/d582f578069b48a598a32779a3195eeb] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp, totalSize=60.2 K 2023-04-18 18:54:35,550 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 795fd43d95ca47609a61262b9cc4132b, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=1217, earliestPutTs=1681844067348 2023-04-18 18:54:35,551 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 3e91ed35ad77414a9b056c7ababf5053, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1237, earliestPutTs=1681844068400 2023-04-18 18:54:35,551 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 76f616c7d80644259d11d230756fa213, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1252, earliestPutTs=1681844070540 2023-04-18 18:54:35,552 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 28b1a650ccce4f369872f8f26bc4894a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1272, earliestPutTs=1681844071203 2023-04-18 18:54:35,552 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting d582f578069b48a598a32779a3195eeb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1285, earliestPutTs=1681844072339 2023-04-18 18:54:35,561 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): cbf5abd278bb2be3a248e6824371aa01#C#compaction#190 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:35,562 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/b2b4edc92814410db40ecb24f47ac279 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/b2b4edc92814410db40ecb24f47ac279 2023-04-18 18:54:35,567 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in cbf5abd278bb2be3a248e6824371aa01/A of cbf5abd278bb2be3a248e6824371aa01 into b2b4edc92814410db40ecb24f47ac279(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:35,567 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:35,567 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., storeName=cbf5abd278bb2be3a248e6824371aa01/A, priority=11, startTime=1681844075111; duration=0sec 2023-04-18 18:54:35,567 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:35,639 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-18 18:54:35,640 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cbf5abd278bb2be3a248e6824371aa01, UNASSIGN}] 2023-04-18 18:54:35,642 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cbf5abd278bb2be3a248e6824371aa01, UNASSIGN 2023-04-18 18:54:35,642 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=cbf5abd278bb2be3a248e6824371aa01, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:35,642 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844075642"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681844075642"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681844075642"}]},"ts":"1681844075642"} 2023-04-18 18:54:35,643 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952}] 2023-04-18 18:54:35,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-18 18:54:35,795 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:35,795 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing cbf5abd278bb2be3a248e6824371aa01, disabling compactions & flushes 2023-04-18 18:54:35,795 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 1 compactions to complete for region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:35,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-18 18:54:35,970 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/eb763ee8220548eb9882c313e48c2649 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/eb763ee8220548eb9882c313e48c2649 2023-04-18 18:54:35,975 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in cbf5abd278bb2be3a248e6824371aa01/C of cbf5abd278bb2be3a248e6824371aa01 into eb763ee8220548eb9882c313e48c2649(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:35,975 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:35,975 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:35,975 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01., storeName=cbf5abd278bb2be3a248e6824371aa01/C, priority=11, startTime=1681844075111; duration=0sec 2023-04-18 18:54:35,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:35,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. after waiting 0 ms 2023-04-18 18:54:35,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:35,975 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:35,975 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing cbf5abd278bb2be3a248e6824371aa01 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-18 18:54:35,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=A 2023-04-18 18:54:35,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:35,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=B 2023-04-18 18:54:35,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:35,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cbf5abd278bb2be3a248e6824371aa01, store=C 2023-04-18 18:54:35,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:36,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-18 18:54:36,387 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418ba135b7c31ad4bac8b824e69cc45b372 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418ba135b7c31ad4bac8b824e69cc45b372 2023-04-18 18:54:36,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-18 18:54:36,792 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1295, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/23df3363debc499aa006111c5872213c 2023-04-18 18:54:37,202 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1295 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/0f7d8bbed7f24ed6b1cb7ce0f39257d0 2023-04-18 18:54:37,612 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1295 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/622d83e30bd34dc7bb92de2192ae646e 2023-04-18 18:54:37,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/A/23df3363debc499aa006111c5872213c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/23df3363debc499aa006111c5872213c 2023-04-18 18:54:37,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/23df3363debc499aa006111c5872213c, entries=100, sequenceid=1295, filesize=18.6 K 2023-04-18 18:54:37,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/B/0f7d8bbed7f24ed6b1cb7ce0f39257d0 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/0f7d8bbed7f24ed6b1cb7ce0f39257d0 2023-04-18 18:54:37,624 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/0f7d8bbed7f24ed6b1cb7ce0f39257d0, entries=100, sequenceid=1295, filesize=9.5 K 2023-04-18 18:54:37,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/.tmp/C/622d83e30bd34dc7bb92de2192ae646e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/622d83e30bd34dc7bb92de2192ae646e 2023-04-18 18:54:37,628 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/622d83e30bd34dc7bb92de2192ae646e, entries=100, sequenceid=1295, filesize=9.5 K 2023-04-18 18:54:37,629 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for cbf5abd278bb2be3a248e6824371aa01 in 1654ms, sequenceid=1295, compaction requested=false 2023-04-18 18:54:37,649 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/21e44d0ec751480780a1106dace9fade, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/225aa18776e042c0b9bba0b5a5e15822, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/99837c2234fd48f49150539f64d97907, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/68a6066a143a47d5ad7ee6026a21309a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/4378f2e643d94e8188ede4a98c3796eb, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/97234ea094904e52aba1596f22b2f652, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/79acfe2c6e854ea2a43f219bb6faf9f0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/be29c06fbf1f4df9b966a68b73d54836, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/2e3c0c072c1645b4a8547b46b6d21c7a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/3357de25dcc2400aab3824afee4f5a6b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/fb95d579ce9f4593be4ba7426e8d89de, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/e7a20fd9aea344ac8513df25797352a9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/c81ada1ff9e64090906f93ce117e0f72, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/ca26529115ea4844ac4099537708e73d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/076d998a3f474df6967624d83c17e1a2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/83544b2f66d24fc88885a7c0f0f18514, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/87e28cf4e53042f0ae2813e708e168ed, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/cf41c80de3bf46cbbd5d17c44b710551, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/c5aca545f704448abce5ab8b6ae3431c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/d0762f8d0484476498aa0bf541ea37f1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/38f5b5b052ac4daa953956b4edb2ede3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/7d51a1ef36214e7fb03ca734f85034a3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/c2ebc8825a0942e7950c18c6976c036f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/a1aaaf61d0b643ffb00f8eb9aa8b17d9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/e2b999a59df84b6cb03b568598a63ae9] to archive 2023-04-18 18:54:37,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-18 18:54:37,650 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 18:54:37,651 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/21e44d0ec751480780a1106dace9fade to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/21e44d0ec751480780a1106dace9fade 2023-04-18 18:54:37,653 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/225aa18776e042c0b9bba0b5a5e15822 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/225aa18776e042c0b9bba0b5a5e15822 2023-04-18 18:54:37,654 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/99837c2234fd48f49150539f64d97907 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/99837c2234fd48f49150539f64d97907 2023-04-18 18:54:37,655 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/68a6066a143a47d5ad7ee6026a21309a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/68a6066a143a47d5ad7ee6026a21309a 2023-04-18 18:54:37,656 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/4378f2e643d94e8188ede4a98c3796eb to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/4378f2e643d94e8188ede4a98c3796eb 2023-04-18 18:54:37,657 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/97234ea094904e52aba1596f22b2f652 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/97234ea094904e52aba1596f22b2f652 2023-04-18 18:54:37,658 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/79acfe2c6e854ea2a43f219bb6faf9f0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/79acfe2c6e854ea2a43f219bb6faf9f0 2023-04-18 18:54:37,659 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/be29c06fbf1f4df9b966a68b73d54836 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/be29c06fbf1f4df9b966a68b73d54836 2023-04-18 18:54:37,659 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/2e3c0c072c1645b4a8547b46b6d21c7a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/2e3c0c072c1645b4a8547b46b6d21c7a 2023-04-18 18:54:37,660 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/3357de25dcc2400aab3824afee4f5a6b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/3357de25dcc2400aab3824afee4f5a6b 2023-04-18 18:54:37,661 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/fb95d579ce9f4593be4ba7426e8d89de to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/fb95d579ce9f4593be4ba7426e8d89de 2023-04-18 18:54:37,662 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/e7a20fd9aea344ac8513df25797352a9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/e7a20fd9aea344ac8513df25797352a9 2023-04-18 18:54:37,663 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/c81ada1ff9e64090906f93ce117e0f72 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/c81ada1ff9e64090906f93ce117e0f72 2023-04-18 18:54:37,665 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/ca26529115ea4844ac4099537708e73d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/ca26529115ea4844ac4099537708e73d 2023-04-18 18:54:37,666 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/076d998a3f474df6967624d83c17e1a2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/076d998a3f474df6967624d83c17e1a2 2023-04-18 18:54:37,667 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/83544b2f66d24fc88885a7c0f0f18514 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/83544b2f66d24fc88885a7c0f0f18514 2023-04-18 18:54:37,668 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/87e28cf4e53042f0ae2813e708e168ed to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/87e28cf4e53042f0ae2813e708e168ed 2023-04-18 18:54:37,669 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/cf41c80de3bf46cbbd5d17c44b710551 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/cf41c80de3bf46cbbd5d17c44b710551 2023-04-18 18:54:37,670 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/c5aca545f704448abce5ab8b6ae3431c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/c5aca545f704448abce5ab8b6ae3431c 2023-04-18 18:54:37,671 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/d0762f8d0484476498aa0bf541ea37f1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/d0762f8d0484476498aa0bf541ea37f1 2023-04-18 18:54:37,672 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/38f5b5b052ac4daa953956b4edb2ede3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/38f5b5b052ac4daa953956b4edb2ede3 2023-04-18 18:54:37,673 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/7d51a1ef36214e7fb03ca734f85034a3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/7d51a1ef36214e7fb03ca734f85034a3 2023-04-18 18:54:37,674 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/c2ebc8825a0942e7950c18c6976c036f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/c2ebc8825a0942e7950c18c6976c036f 2023-04-18 18:54:37,675 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/a1aaaf61d0b643ffb00f8eb9aa8b17d9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/a1aaaf61d0b643ffb00f8eb9aa8b17d9 2023-04-18 18:54:37,676 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/e2b999a59df84b6cb03b568598a63ae9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/e2b999a59df84b6cb03b568598a63ae9 2023-04-18 18:54:37,678 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/e78eda01e3834dc89c4c6696a487d30a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/161fd02fb99447dea3002ef9fbf41dee, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/225ef43b3d884550ac9c6c9ade3e0a2e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/aea431548e3646b69982c165eb2532be, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/176501c0f2d040fc8f0ee73d8588f1b3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/27365794d38f469186d7ee8e373f0bc4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/894b8e318d214f18b9bf0ed96bc87077, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/653b8766284d436dbc1970b8448bdecb, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/2dbc95602be54157a186710e92fbb49c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/b74de95db2794a98acc34eb32b22d1ef, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/ee2b2ffdfa4144a995756dd2f012765b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/67be30e6125c4106be54fb0691433b7a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/0021b1b233394ded90cdf8792287c6f3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/62d845575eb548659248f3c82884665c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/bc42c0e5139e4d49901d4f27345f0341, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/3ebf71667e5c4e6bb9d1e6e746c75e07, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/55644aba3c284cd1b9dda4432c689ec3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/b0d0eff462374d3bb84827d45b6c6575, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/ed69ec5f64d34f688aa0ab206c66a485, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/88cdefb4bae14576a29e4fd5420621fc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/b141b33cb4eb4621b8ad5c9ef0669c61, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/8a00f6e2f02d442b8934460012ea6617, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/cd37f1578ba344438cf181a997ef769d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/50f442b6204c41a19ff0955e8f4319a0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/3f0dd9c329dd4ecb94dfd74a202bfb08] to archive 2023-04-18 18:54:37,679 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 18:54:37,680 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/e78eda01e3834dc89c4c6696a487d30a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/e78eda01e3834dc89c4c6696a487d30a 2023-04-18 18:54:37,681 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/161fd02fb99447dea3002ef9fbf41dee to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/161fd02fb99447dea3002ef9fbf41dee 2023-04-18 18:54:37,682 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/225ef43b3d884550ac9c6c9ade3e0a2e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/225ef43b3d884550ac9c6c9ade3e0a2e 2023-04-18 18:54:37,683 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/aea431548e3646b69982c165eb2532be to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/aea431548e3646b69982c165eb2532be 2023-04-18 18:54:37,684 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/176501c0f2d040fc8f0ee73d8588f1b3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/176501c0f2d040fc8f0ee73d8588f1b3 2023-04-18 18:54:37,685 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/27365794d38f469186d7ee8e373f0bc4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/27365794d38f469186d7ee8e373f0bc4 2023-04-18 18:54:37,687 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/894b8e318d214f18b9bf0ed96bc87077 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/894b8e318d214f18b9bf0ed96bc87077 2023-04-18 18:54:37,687 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/653b8766284d436dbc1970b8448bdecb to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/653b8766284d436dbc1970b8448bdecb 2023-04-18 18:54:37,689 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/2dbc95602be54157a186710e92fbb49c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/2dbc95602be54157a186710e92fbb49c 2023-04-18 18:54:37,690 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/b74de95db2794a98acc34eb32b22d1ef to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/b74de95db2794a98acc34eb32b22d1ef 2023-04-18 18:54:37,690 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/ee2b2ffdfa4144a995756dd2f012765b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/ee2b2ffdfa4144a995756dd2f012765b 2023-04-18 18:54:37,691 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/67be30e6125c4106be54fb0691433b7a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/67be30e6125c4106be54fb0691433b7a 2023-04-18 18:54:37,692 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/0021b1b233394ded90cdf8792287c6f3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/0021b1b233394ded90cdf8792287c6f3 2023-04-18 18:54:37,693 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/62d845575eb548659248f3c82884665c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/62d845575eb548659248f3c82884665c 2023-04-18 18:54:37,694 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/bc42c0e5139e4d49901d4f27345f0341 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/bc42c0e5139e4d49901d4f27345f0341 2023-04-18 18:54:37,695 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/3ebf71667e5c4e6bb9d1e6e746c75e07 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/3ebf71667e5c4e6bb9d1e6e746c75e07 2023-04-18 18:54:37,697 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/55644aba3c284cd1b9dda4432c689ec3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/55644aba3c284cd1b9dda4432c689ec3 2023-04-18 18:54:37,698 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/b0d0eff462374d3bb84827d45b6c6575 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/b0d0eff462374d3bb84827d45b6c6575 2023-04-18 18:54:37,699 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/ed69ec5f64d34f688aa0ab206c66a485 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/ed69ec5f64d34f688aa0ab206c66a485 2023-04-18 18:54:37,700 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/88cdefb4bae14576a29e4fd5420621fc to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/88cdefb4bae14576a29e4fd5420621fc 2023-04-18 18:54:37,701 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/b141b33cb4eb4621b8ad5c9ef0669c61 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/b141b33cb4eb4621b8ad5c9ef0669c61 2023-04-18 18:54:37,702 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/8a00f6e2f02d442b8934460012ea6617 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/8a00f6e2f02d442b8934460012ea6617 2023-04-18 18:54:37,703 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/cd37f1578ba344438cf181a997ef769d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/cd37f1578ba344438cf181a997ef769d 2023-04-18 18:54:37,704 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/50f442b6204c41a19ff0955e8f4319a0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/50f442b6204c41a19ff0955e8f4319a0 2023-04-18 18:54:37,704 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/3f0dd9c329dd4ecb94dfd74a202bfb08 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/3f0dd9c329dd4ecb94dfd74a202bfb08 2023-04-18 18:54:37,707 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/708aeb4509414ca2b022536786c9dd80, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/d405f7c6c47848cfa990d6d4483503ab, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/219231f16fa04e51950b75f1bb5def8b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/dc8d1b54b39840aa958407bdf20342b3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/20f130deb76344ad9f0bf0e979848848, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/b0ebce8ccfa748f5acda15bc88ab73f2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/42414bc697ad4899bbd86cdab462c9fc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/a4f91e1665c040b1afa572d2100a98f0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/577d83a5163d489d960cc06957720521, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/0cc3d3b3525d4098ad1e2b52c0552bb6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/7ad8c3b0369e43a290f387d76fd30498, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/276135bd3cf744f8966f7c2dccc1cd55, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/8d202aa3c1e04bb4a23709fbaf8b4184, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/65a2bb22879d4f20ac3ddfff949699c6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/97c3768a16a34c6dbeccfecd397bb7cd, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/c6f2cc26137e4aec8e78a6b007a1ae6c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/07c5b76b4cb84bd789128cde98708425, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/9ea95ecf15574cedb4655a2009f18ab9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/9083fe56abf04dca9b9817aeaca9fe8a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/795fd43d95ca47609a61262b9cc4132b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/b0c245d0ac3c43f687ac257a3b6416a6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/3e91ed35ad77414a9b056c7ababf5053, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/76f616c7d80644259d11d230756fa213, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/28b1a650ccce4f369872f8f26bc4894a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/d582f578069b48a598a32779a3195eeb] to archive 2023-04-18 18:54:37,707 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 18:54:37,709 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/708aeb4509414ca2b022536786c9dd80 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/708aeb4509414ca2b022536786c9dd80 2023-04-18 18:54:37,710 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/d405f7c6c47848cfa990d6d4483503ab to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/d405f7c6c47848cfa990d6d4483503ab 2023-04-18 18:54:37,711 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/219231f16fa04e51950b75f1bb5def8b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/219231f16fa04e51950b75f1bb5def8b 2023-04-18 18:54:37,711 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/dc8d1b54b39840aa958407bdf20342b3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/dc8d1b54b39840aa958407bdf20342b3 2023-04-18 18:54:37,712 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/20f130deb76344ad9f0bf0e979848848 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/20f130deb76344ad9f0bf0e979848848 2023-04-18 18:54:37,713 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/b0ebce8ccfa748f5acda15bc88ab73f2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/b0ebce8ccfa748f5acda15bc88ab73f2 2023-04-18 18:54:37,714 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/42414bc697ad4899bbd86cdab462c9fc to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/42414bc697ad4899bbd86cdab462c9fc 2023-04-18 18:54:37,715 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/a4f91e1665c040b1afa572d2100a98f0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/a4f91e1665c040b1afa572d2100a98f0 2023-04-18 18:54:37,716 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/577d83a5163d489d960cc06957720521 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/577d83a5163d489d960cc06957720521 2023-04-18 18:54:37,717 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/0cc3d3b3525d4098ad1e2b52c0552bb6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/0cc3d3b3525d4098ad1e2b52c0552bb6 2023-04-18 18:54:37,718 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/7ad8c3b0369e43a290f387d76fd30498 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/7ad8c3b0369e43a290f387d76fd30498 2023-04-18 18:54:37,719 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/276135bd3cf744f8966f7c2dccc1cd55 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/276135bd3cf744f8966f7c2dccc1cd55 2023-04-18 18:54:37,720 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/8d202aa3c1e04bb4a23709fbaf8b4184 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/8d202aa3c1e04bb4a23709fbaf8b4184 2023-04-18 18:54:37,721 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/65a2bb22879d4f20ac3ddfff949699c6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/65a2bb22879d4f20ac3ddfff949699c6 2023-04-18 18:54:37,722 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/97c3768a16a34c6dbeccfecd397bb7cd to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/97c3768a16a34c6dbeccfecd397bb7cd 2023-04-18 18:54:37,723 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/c6f2cc26137e4aec8e78a6b007a1ae6c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/c6f2cc26137e4aec8e78a6b007a1ae6c 2023-04-18 18:54:37,724 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/07c5b76b4cb84bd789128cde98708425 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/07c5b76b4cb84bd789128cde98708425 2023-04-18 18:54:37,725 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/9ea95ecf15574cedb4655a2009f18ab9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/9ea95ecf15574cedb4655a2009f18ab9 2023-04-18 18:54:37,725 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/9083fe56abf04dca9b9817aeaca9fe8a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/9083fe56abf04dca9b9817aeaca9fe8a 2023-04-18 18:54:37,726 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/795fd43d95ca47609a61262b9cc4132b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/795fd43d95ca47609a61262b9cc4132b 2023-04-18 18:54:37,727 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/b0c245d0ac3c43f687ac257a3b6416a6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/b0c245d0ac3c43f687ac257a3b6416a6 2023-04-18 18:54:37,728 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/3e91ed35ad77414a9b056c7ababf5053 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/3e91ed35ad77414a9b056c7ababf5053 2023-04-18 18:54:37,729 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/76f616c7d80644259d11d230756fa213 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/76f616c7d80644259d11d230756fa213 2023-04-18 18:54:37,730 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/28b1a650ccce4f369872f8f26bc4894a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/28b1a650ccce4f369872f8f26bc4894a 2023-04-18 18:54:37,731 DEBUG [StoreCloser-TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/d582f578069b48a598a32779a3195eeb to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/d582f578069b48a598a32779a3195eeb 2023-04-18 18:54:37,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/recovered.edits/1298.seqid, newMaxSeqId=1298, maxSeqId=4 2023-04-18 18:54:37,737 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01. 2023-04-18 18:54:37,737 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for cbf5abd278bb2be3a248e6824371aa01: 2023-04-18 18:54:37,738 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:37,738 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=cbf5abd278bb2be3a248e6824371aa01, regionState=CLOSED 2023-04-18 18:54:37,739 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844077738"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681844077738"}]},"ts":"1681844077738"} 2023-04-18 18:54:37,741 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-18 18:54:37,741 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure cbf5abd278bb2be3a248e6824371aa01, server=jenkins-hbase4.apache.org,34053,1681844018952 in 2.0960 sec 2023-04-18 18:54:37,742 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-18 18:54:37,742 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=cbf5abd278bb2be3a248e6824371aa01, UNASSIGN in 2.1010 sec 2023-04-18 18:54:37,743 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844077742"}]},"ts":"1681844077742"} 2023-04-18 18:54:37,743 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-18 18:54:37,745 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-18 18:54:37,746 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.2070 sec 2023-04-18 18:54:39,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-18 18:54:39,650 INFO [Listener at localhost/36951] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-18 18:54:39,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-18 18:54:39,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:54:39,653 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:54:39,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-18 18:54:39,654 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:54:39,658 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:39,660 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A, FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B, FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C, FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/recovered.edits] 2023-04-18 18:54:39,663 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/23df3363debc499aa006111c5872213c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/23df3363debc499aa006111c5872213c 2023-04-18 18:54:39,664 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/b2b4edc92814410db40ecb24f47ac279 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/A/b2b4edc92814410db40ecb24f47ac279 2023-04-18 18:54:39,666 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/0f7d8bbed7f24ed6b1cb7ce0f39257d0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/0f7d8bbed7f24ed6b1cb7ce0f39257d0 2023-04-18 18:54:39,667 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/7a009b6083ce4bfeb51572452325ed3a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/B/7a009b6083ce4bfeb51572452325ed3a 2023-04-18 18:54:39,669 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/622d83e30bd34dc7bb92de2192ae646e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/622d83e30bd34dc7bb92de2192ae646e 2023-04-18 18:54:39,670 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/eb763ee8220548eb9882c313e48c2649 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/C/eb763ee8220548eb9882c313e48c2649 2023-04-18 18:54:39,673 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/recovered.edits/1298.seqid to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01/recovered.edits/1298.seqid 2023-04-18 18:54:39,674 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/cbf5abd278bb2be3a248e6824371aa01 2023-04-18 18:54:39,674 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 18:54:39,674 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-18 18:54:39,675 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-18 18:54:39,678 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304180a6b8e0446f2488f8c7bd17f6da740d3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304180a6b8e0446f2488f8c7bd17f6da740d3 2023-04-18 18:54:39,679 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041811935a4d29e04ceb8c1ac1f7454bff5e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041811935a4d29e04ceb8c1ac1f7454bff5e 2023-04-18 18:54:39,680 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304181a58efce06a74f96bfa9e3ff986ef621 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304181a58efce06a74f96bfa9e3ff986ef621 2023-04-18 18:54:39,681 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304181de7af9c9b33451cbf2f041dc47bb10b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304181de7af9c9b33451cbf2f041dc47bb10b 2023-04-18 18:54:39,682 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304181ea46204c5bd48a28bce372228bd1a21 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304181ea46204c5bd48a28bce372228bd1a21 2023-04-18 18:54:39,683 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041820da66c283c84ae89b405e7110c31bca to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041820da66c283c84ae89b405e7110c31bca 2023-04-18 18:54:39,684 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418368ed5f857714b8f96c1dc1691c8cf82 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418368ed5f857714b8f96c1dc1691c8cf82 2023-04-18 18:54:39,685 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418369b7fc552ea46b98da3563cafaa98d2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418369b7fc552ea46b98da3563cafaa98d2 2023-04-18 18:54:39,686 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304184587a77d83a24cad8b61df6ad2793f2b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304184587a77d83a24cad8b61df6ad2793f2b 2023-04-18 18:54:39,688 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041850a946cb3c3d45f18d6edc73aa9525e2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041850a946cb3c3d45f18d6edc73aa9525e2 2023-04-18 18:54:39,689 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418612bbe612d8c4b2092d7a38622dfcd7b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418612bbe612d8c4b2092d7a38622dfcd7b 2023-04-18 18:54:39,690 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304186b6c2a799fac4da1a23a922f48d587ac to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304186b6c2a799fac4da1a23a922f48d587ac 2023-04-18 18:54:39,692 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418728fd867af624b86bf3596757bb18bc3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418728fd867af624b86bf3596757bb18bc3 2023-04-18 18:54:39,693 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304188411bfa393cf4b3e84a0c13ce5491258 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304188411bfa393cf4b3e84a0c13ce5491258 2023-04-18 18:54:39,694 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304189a7f9a2af25b47f89d9904c44c419e12 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304189a7f9a2af25b47f89d9904c44c419e12 2023-04-18 18:54:39,696 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a19570c1167e48709d6ae45db43df48e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a19570c1167e48709d6ae45db43df48e 2023-04-18 18:54:39,697 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a1cb61b378704beb97b7d3fcac90f54a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a1cb61b378704beb97b7d3fcac90f54a 2023-04-18 18:54:39,698 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418ba135b7c31ad4bac8b824e69cc45b372 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418ba135b7c31ad4bac8b824e69cc45b372 2023-04-18 18:54:39,699 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418c5b520703f4a4021b7a3c065602dc350 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418c5b520703f4a4021b7a3c065602dc350 2023-04-18 18:54:39,701 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418ca019b76dd5e4e6ba31ce54fa0502744 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418ca019b76dd5e4e6ba31ce54fa0502744 2023-04-18 18:54:39,702 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418d652429c2a7b45169e1ea231a55d8898 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418d652429c2a7b45169e1ea231a55d8898 2023-04-18 18:54:39,704 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e60525ab5eb045b9a9d809dcb918a865 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e60525ab5eb045b9a9d809dcb918a865 2023-04-18 18:54:39,704 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-18 18:54:39,712 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:54:39,715 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-18 18:54:39,718 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-18 18:54:39,725 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:54:39,725 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-18 18:54:39,725 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681844079725"}]},"ts":"9223372036854775807"} 2023-04-18 18:54:39,734 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-18 18:54:39,734 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => cbf5abd278bb2be3a248e6824371aa01, NAME => 'TestAcidGuarantees,,1681844049657.cbf5abd278bb2be3a248e6824371aa01.', STARTKEY => '', ENDKEY => ''}] 2023-04-18 18:54:39,734 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-18 18:54:39,734 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681844079734"}]},"ts":"9223372036854775807"} 2023-04-18 18:54:39,736 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-18 18:54:39,739 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:54:39,740 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 88 msec 2023-04-18 18:54:39,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-18 18:54:39,755 INFO [Listener at localhost/36951] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-18 18:54:39,769 INFO [Listener at localhost/36951] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=232 (was 235), OpenFileDescriptor=553 (was 551) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=378 (was 377) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 170), AvailableMemoryMB=6615 (was 6852) 2023-04-18 18:54:39,781 INFO [Listener at localhost/36951] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=232, OpenFileDescriptor=553, MaxFileDescriptor=60000, SystemLoadAverage=378, ProcessCount=169, AvailableMemoryMB=6615 2023-04-18 18:54:39,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-18 18:54:39,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-18 18:54:39,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-18 18:54:39,786 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-18 18:54:39,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-18 18:54:39,787 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-18 18:54:39,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-18 18:54:39,789 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:39,789 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a empty. 2023-04-18 18:54:39,790 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:39,790 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 18:54:39,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-18 18:54:40,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-18 18:54:40,199 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-18 18:54:40,200 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => e3c3c3367729f1159cdc97339f70376a, NAME => 'TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp 2023-04-18 18:54:40,287 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-18 18:54:40,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-18 18:54:40,606 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 18:54:40,606 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing e3c3c3367729f1159cdc97339f70376a, disabling compactions & flushes 2023-04-18 18:54:40,606 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:40,606 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:40,606 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. after waiting 0 ms 2023-04-18 18:54:40,607 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:40,607 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:40,607 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:40,609 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-18 18:54:40,610 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844080609"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681844080609"}]},"ts":"1681844080609"} 2023-04-18 18:54:40,610 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-18 18:54:40,611 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-18 18:54:40,611 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844080611"}]},"ts":"1681844080611"} 2023-04-18 18:54:40,612 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-18 18:54:40,615 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e3c3c3367729f1159cdc97339f70376a, ASSIGN}] 2023-04-18 18:54:40,617 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e3c3c3367729f1159cdc97339f70376a, ASSIGN 2023-04-18 18:54:40,617 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e3c3c3367729f1159cdc97339f70376a, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34053,1681844018952; forceNewPlan=false, retain=false 2023-04-18 18:54:40,769 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=e3c3c3367729f1159cdc97339f70376a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:40,769 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844080768"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681844080768"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681844080768"}]},"ts":"1681844080768"} 2023-04-18 18:54:40,770 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952}] 2023-04-18 18:54:40,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-18 18:54:40,927 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:40,927 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e3c3c3367729f1159cdc97339f70376a, NAME => 'TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.', STARTKEY => '', ENDKEY => ''} 2023-04-18 18:54:40,927 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:40,927 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 18:54:40,927 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:40,927 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:40,929 INFO [StoreOpener-e3c3c3367729f1159cdc97339f70376a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:40,931 DEBUG [StoreOpener-e3c3c3367729f1159cdc97339f70376a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A 2023-04-18 18:54:40,931 DEBUG [StoreOpener-e3c3c3367729f1159cdc97339f70376a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A 2023-04-18 18:54:40,931 INFO [StoreOpener-e3c3c3367729f1159cdc97339f70376a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:54:40,932 INFO [StoreOpener-e3c3c3367729f1159cdc97339f70376a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e3c3c3367729f1159cdc97339f70376a columnFamilyName A 2023-04-18 18:54:40,932 INFO [StoreOpener-e3c3c3367729f1159cdc97339f70376a-1] regionserver.HStore(310): Store=e3c3c3367729f1159cdc97339f70376a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:54:40,932 INFO [StoreOpener-e3c3c3367729f1159cdc97339f70376a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:40,934 DEBUG [StoreOpener-e3c3c3367729f1159cdc97339f70376a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B 2023-04-18 18:54:40,934 DEBUG [StoreOpener-e3c3c3367729f1159cdc97339f70376a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B 2023-04-18 18:54:40,934 INFO [StoreOpener-e3c3c3367729f1159cdc97339f70376a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:54:40,934 INFO [StoreOpener-e3c3c3367729f1159cdc97339f70376a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e3c3c3367729f1159cdc97339f70376a columnFamilyName B 2023-04-18 18:54:40,935 INFO [StoreOpener-e3c3c3367729f1159cdc97339f70376a-1] regionserver.HStore(310): Store=e3c3c3367729f1159cdc97339f70376a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:54:40,935 INFO [StoreOpener-e3c3c3367729f1159cdc97339f70376a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:40,936 DEBUG [StoreOpener-e3c3c3367729f1159cdc97339f70376a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C 2023-04-18 18:54:40,936 DEBUG [StoreOpener-e3c3c3367729f1159cdc97339f70376a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C 2023-04-18 18:54:40,936 INFO [StoreOpener-e3c3c3367729f1159cdc97339f70376a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:54:40,937 INFO [StoreOpener-e3c3c3367729f1159cdc97339f70376a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e3c3c3367729f1159cdc97339f70376a columnFamilyName C 2023-04-18 18:54:40,937 INFO [StoreOpener-e3c3c3367729f1159cdc97339f70376a-1] regionserver.HStore(310): Store=e3c3c3367729f1159cdc97339f70376a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:54:40,937 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:40,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:40,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:40,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 18:54:40,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:40,943 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-18 18:54:40,943 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e3c3c3367729f1159cdc97339f70376a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67303197, jitterRate=0.0028957873582839966}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 18:54:40,943 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:40,944 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., pid=30, masterSystemTime=1681844080921 2023-04-18 18:54:40,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:40,945 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:40,946 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=e3c3c3367729f1159cdc97339f70376a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:40,946 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844080946"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681844080946"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681844080946"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681844080946"}]},"ts":"1681844080946"} 2023-04-18 18:54:40,948 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-18 18:54:40,948 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 in 177 msec 2023-04-18 18:54:40,950 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-18 18:54:40,950 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e3c3c3367729f1159cdc97339f70376a, ASSIGN in 333 msec 2023-04-18 18:54:40,950 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-18 18:54:40,951 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844080951"}]},"ts":"1681844080951"} 2023-04-18 18:54:40,951 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-18 18:54:40,953 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-18 18:54:40,954 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1700 sec 2023-04-18 18:54:41,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-18 18:54:41,892 INFO [Listener at localhost/36951] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-18 18:54:41,893 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x2d7cdc2b to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:54:41,899 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5b4a16d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:54:41,900 DEBUG [Listener at localhost/36951] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:54:41,901 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36676, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:54:41,902 DEBUG [Listener at localhost/36951] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-18 18:54:41,904 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50900, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-18 18:54:41,905 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x5096c69b to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:54:41,911 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b40c526, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:54:41,911 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x216aad74 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:54:41,916 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@47ae15dc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:54:41,917 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x76203061 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:54:41,920 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25522dc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:54:41,921 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x3b39d5b8 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:54:41,924 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4c6d5986, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:54:41,925 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x44f9bd75 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:54:41,929 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31182447, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:54:41,929 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x574000bf to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:54:41,933 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@270a3c0c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:54:41,933 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x4994df89 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:54:41,940 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e067bfc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:54:41,941 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x69bc154a to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:54:41,944 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6bbe763, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:54:41,944 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x15186778 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:54:41,949 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c8736d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:54:41,949 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x13e90e8d to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:54:41,952 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@451c882f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:54:41,956 DEBUG [hconnection-0x2ea4df52-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:54:41,957 DEBUG [hconnection-0x783449dc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:54:41,957 DEBUG [hconnection-0x2609cc90-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:54:41,957 DEBUG [hconnection-0x1998d18f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:54:41,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:41,958 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36712, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:54:41,960 DEBUG [hconnection-0x7ad2e23e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:54:41,960 DEBUG [hconnection-0x2de25b08-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:54:41,960 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36686, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:54:41,960 DEBUG [hconnection-0xebdadfe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:54:41,960 DEBUG [hconnection-0x417cfcfd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:54:41,966 DEBUG [hconnection-0x45dc9b18-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:54:41,966 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36688, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:54:41,966 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36696, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:54:41,966 DEBUG [hconnection-0x3c333ca0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:54:41,968 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36732, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:54:41,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:41,968 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36740, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:54:41,969 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36772, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:54:41,968 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36756, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:54:41,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:41,970 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:41,970 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:41,970 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:41,970 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:41,970 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36720, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:54:41,971 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:41,971 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:41,973 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:41,973 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:41,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:41,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:41,973 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:41,973 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:41,975 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36776, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:54:41,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:41,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:41,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:41,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:41,982 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:41,983 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:41,983 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:41,984 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:41,984 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:41,985 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-18 18:54:41,985 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:41,986 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:41,986 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:41,986 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:41,986 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:41,986 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:42,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:42,020 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:42,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844081988, max timestamp=1681844082035Num uniques 859; 2023-04-18 18:54:42,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681844081988, max timestamp=1681844082035Num uniques 859; 2023-04-18 18:54:42,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844081988, max timestamp=1681844082035Num uniques 859; 2023-04-18 18:54:42,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-18 18:54:42,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:54:42,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:54:42,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:42,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36772 deadline: 1681844142046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:42,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36696 deadline: 1681844142047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681844082045, max timestamp=1681844082049Num uniques 1009; 2023-04-18 18:54:42,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=9978, suffix heap size=132904,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=218 2023-04-18 18:54:42,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681844082045, max timestamp=1681844082049Num uniques 1018; 2023-04-18 18:54:42,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-18 18:54:42,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681844082045, max timestamp=1681844082049Num uniques 1018; 2023-04-18 18:54:42,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-18 18:54:42,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:42,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844142067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:42,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844142068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:42,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844142068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:42,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36772 deadline: 1681844142151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:42,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36696 deadline: 1681844142152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:42,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844142170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:42,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844142171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:42,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844142171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:42,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36772 deadline: 1681844142356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:42,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36696 deadline: 1681844142356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:42,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844142373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:42,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844142375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:42,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844142375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,413 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/ded014cc35fa4880ada9724ed3b5158f 2023-04-18 18:54:42,426 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/60de32464a4545e6945720fb7fcfc843 2023-04-18 18:54:42,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:42,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36772 deadline: 1681844142659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:42,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36696 deadline: 1681844142661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:42,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844142678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:42,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844142678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:42,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844142680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,837 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/6cae18f511504c9aa01e981e241d7186 2023-04-18 18:54:42,842 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/ded014cc35fa4880ada9724ed3b5158f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/ded014cc35fa4880ada9724ed3b5158f 2023-04-18 18:54:42,846 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/ded014cc35fa4880ada9724ed3b5158f, entries=150, sequenceid=10, filesize=11.6 K 2023-04-18 18:54:42,847 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/60de32464a4545e6945720fb7fcfc843 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/60de32464a4545e6945720fb7fcfc843 2023-04-18 18:54:42,851 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/60de32464a4545e6945720fb7fcfc843, entries=150, sequenceid=10, filesize=11.6 K 2023-04-18 18:54:42,851 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/6cae18f511504c9aa01e981e241d7186 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/6cae18f511504c9aa01e981e241d7186 2023-04-18 18:54:42,855 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/6cae18f511504c9aa01e981e241d7186, entries=150, sequenceid=10, filesize=11.6 K 2023-04-18 18:54:42,856 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for e3c3c3367729f1159cdc97339f70376a in 871ms, sequenceid=10, compaction requested=false 2023-04-18 18:54:42,856 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-18 18:54:42,856 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:42,856 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:42,856 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:42,857 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:42,857 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,857 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:42,857 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:42,862 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,862 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:42,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:42,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:42,862 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:42,862 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:42,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:42,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:42,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:42,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:42,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:42,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58f44fc6[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:42,864 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:42,864 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:42,865 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:42,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:42,865 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:42,865 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:42,865 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:42,865 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,865 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:42,867 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,867 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:42,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:42,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:42,867 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:42,867 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:42,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:42,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:42,870 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:42,870 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:42,870 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:42,870 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:42,870 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:42,873 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:42,873 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:42,873 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:42,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:42,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:42,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:42,874 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:42,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:42,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:42,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:42,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:42,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:42,875 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:42,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:42,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,892 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:42,892 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:42,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:42,892 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:42,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:42,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:42,892 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:42,892 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:42,892 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:42,892 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,892 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:42,892 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:42,892 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:42,892 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:42,892 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:42,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:42,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:42,993 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:42,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:42,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:42,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:42,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:42,997 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:42,997 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:42,997 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:42,997 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:42,997 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:42,997 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:42,998 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:42,998 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:42,999 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:42,999 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:42,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:43,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:43,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:43,000 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:43,000 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:43,000 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:43,000 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:43,001 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:43,001 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:43,001 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:43,001 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=140.89 KB heapSize=399.38 KB 2023-04-18 18:54:43,001 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:43,001 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:43,001 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:43,001 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:43,001 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:43,001 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:43,010 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/4a295d84e69f4ba09f035db3fe13c51e 2023-04-18 18:54:43,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:43,165 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:43,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:43,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36696 deadline: 1681844143171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:43,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36772 deadline: 1681844143173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:43,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844143182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:43,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844143184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:43,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844143184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,266 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-18 18:54:43,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:43,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36696 deadline: 1681844143274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:43,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36772 deadline: 1681844143276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,422 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/0a230ea48e6b417db6923b93090721c8 2023-04-18 18:54:43,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:43,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36696 deadline: 1681844143479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,483 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:43,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36772 deadline: 1681844143481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:43,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36696 deadline: 1681844143782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:43,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36772 deadline: 1681844143786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,836 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/fd89a62a7bfa4b758ef02e83f286009d 2023-04-18 18:54:43,841 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/4a295d84e69f4ba09f035db3fe13c51e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/4a295d84e69f4ba09f035db3fe13c51e 2023-04-18 18:54:43,845 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/4a295d84e69f4ba09f035db3fe13c51e, entries=150, sequenceid=64, filesize=11.6 K 2023-04-18 18:54:43,846 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/0a230ea48e6b417db6923b93090721c8 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/0a230ea48e6b417db6923b93090721c8 2023-04-18 18:54:43,850 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/0a230ea48e6b417db6923b93090721c8, entries=150, sequenceid=64, filesize=11.6 K 2023-04-18 18:54:43,851 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/fd89a62a7bfa4b758ef02e83f286009d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/fd89a62a7bfa4b758ef02e83f286009d 2023-04-18 18:54:43,855 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/fd89a62a7bfa4b758ef02e83f286009d, entries=150, sequenceid=64, filesize=11.6 K 2023-04-18 18:54:43,856 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~399.33 KB/408912, currentSize=40.25 KB/41220 for e3c3c3367729f1159cdc97339f70376a in 855ms, sequenceid=64, compaction requested=false 2023-04-18 18:54:43,856 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:43,856 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:43,856 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:43,856 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:43,856 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,857 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:43,857 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:43,858 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,858 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:43,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:43,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:43,859 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:43,859 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:43,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:43,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:43,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:43,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:43,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:43,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bea1df4[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:43,860 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:43,860 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:43,862 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:43,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:43,863 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:43,863 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:43,863 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:43,863 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,863 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:43,864 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,864 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:43,864 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:43,864 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:43,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:43,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:43,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:43,867 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:43,867 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:43,867 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:43,867 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:43,867 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:43,868 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:43,868 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:43,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:43,868 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:43,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:43,869 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:43,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:43,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:43,871 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,873 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:43,873 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:43,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:43,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:43,874 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:43,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:43,874 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:43,874 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:43,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:43,874 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:43,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:43,874 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:43,874 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:43,874 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,874 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:43,874 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:43,874 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:43,974 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:43,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:43,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:43,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:43,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:43,982 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:43,982 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:43,983 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:43,983 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:43,983 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:43,983 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:43,985 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:43,985 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:43,985 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:43,985 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:43,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:43,986 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:43,986 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:43,986 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:43,986 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:43,986 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:43,986 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:43,987 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:43,987 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:43,987 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:43,987 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:43,987 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-18 18:54:43,987 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:43,987 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:43,987 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:43,987 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:43,987 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:43,987 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:44,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:44,191 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:44,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681844084200, max timestamp=1681844084210Num uniques 859; 2023-04-18 18:54:44,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681844084200, max timestamp=1681844084210Num uniques 859; 2023-04-18 18:54:44,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 18:54:44,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681844084200, max timestamp=1681844084210Num uniques 859; 2023-04-18 18:54:44,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 18:54:44,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 18:54:44,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:44,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844144229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:44,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:44,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:44,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844144229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:44,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844144230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:44,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:44,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36696 deadline: 1681844144285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:44,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:44,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36772 deadline: 1681844144292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:44,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:44,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844144332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:44,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:44,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844144335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:44,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:44,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844144335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:44,398 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=73 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/5a52b1f63e0e4b27a96dca32ba575b9e 2023-04-18 18:54:44,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:44,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844144535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:44,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:44,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844144537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:44,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:44,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844144538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:44,809 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=73 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/8c786d850e68420f9c80cb61b5a8b2ae 2023-04-18 18:54:44,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:44,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844144838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:44,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:44,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844144842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:44,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:44,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844144842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,220 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=73 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/b0917fd341184707be47c165186394df 2023-04-18 18:54:45,225 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/5a52b1f63e0e4b27a96dca32ba575b9e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/5a52b1f63e0e4b27a96dca32ba575b9e 2023-04-18 18:54:45,229 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/5a52b1f63e0e4b27a96dca32ba575b9e, entries=100, sequenceid=73, filesize=9.3 K 2023-04-18 18:54:45,230 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/8c786d850e68420f9c80cb61b5a8b2ae as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/8c786d850e68420f9c80cb61b5a8b2ae 2023-04-18 18:54:45,235 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/8c786d850e68420f9c80cb61b5a8b2ae, entries=100, sequenceid=73, filesize=9.3 K 2023-04-18 18:54:45,236 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/b0917fd341184707be47c165186394df as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b0917fd341184707be47c165186394df 2023-04-18 18:54:45,240 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b0917fd341184707be47c165186394df, entries=100, sequenceid=73, filesize=9.3 K 2023-04-18 18:54:45,241 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for e3c3c3367729f1159cdc97339f70376a in 1254ms, sequenceid=73, compaction requested=true 2023-04-18 18:54:45,241 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:45,241 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:45,241 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:45,241 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:45,241 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,241 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:45,241 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:45,243 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,243 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:45,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:45,244 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,244 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:45,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:45,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ac8c645[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:45,248 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:45,248 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,250 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,250 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:45,250 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:45,250 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:45,250 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,250 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:45,252 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:45,252 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,252 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:45,252 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:45,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:45,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:45,254 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:45,255 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:45,255 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:45,255 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:45,255 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:45,257 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,257 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,257 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:45,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:45,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,258 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,259 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:45,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,262 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:45,262 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,262 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:45,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:45,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:45,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:45,262 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,263 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:45,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,263 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:45,263 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:45,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:45,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:45,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:45,295 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=154.31 KB heapSize=436.08 KB 2023-04-18 18:54:45,295 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:45,295 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 18:54:45,296 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:45,296 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 18:54:45,296 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:45,296 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 18:54:45,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:45,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36772 deadline: 1681844145301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:45,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36696 deadline: 1681844145302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:45,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844145340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:45,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844145344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:45,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844145348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,363 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:45,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:45,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:45,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:45,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:45,367 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:45,367 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:45,367 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:45,367 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:45,368 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,368 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,370 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:45,370 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:45,370 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,370 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:45,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:45,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:45,371 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:45,371 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:45,371 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:45,372 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:45,372 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:45,372 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:45,372 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:45,372 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:45,372 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:45,372 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:45,372 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:45,372 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,372 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:45,372 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:45,374 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,374 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:45,374 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,375 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:45,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:45,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25856ac9[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:45,376 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:45,376 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,382 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,382 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:45,382 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:45,382 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,382 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:45,382 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:45,384 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:45,384 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,384 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:45,384 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:45,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:45,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:45,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:45,386 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:45,386 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:45,386 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:45,386 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:45,386 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:45,388 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,388 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:45,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,388 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:45,389 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,390 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,393 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,393 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:45,393 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:45,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:45,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:45,393 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:45,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:45,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:45,393 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:45,393 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,393 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,393 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,393 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,393 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,394 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:45,394 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:45,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:45,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:45,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36772 deadline: 1681844145405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:45,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36696 deadline: 1681844145406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,494 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:45,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:45,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:45,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:45,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:45,498 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:45,498 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:45,498 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:45,498 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:45,498 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,498 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,499 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,499 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:45,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,500 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,500 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:45,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:45,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:45,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:45,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:45,501 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:45,501 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:45,501 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:45,501 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:45,501 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:45,501 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:45,502 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:45,502 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:45,502 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:45,502 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:45,502 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:45,502 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,502 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:45,502 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:45,504 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,504 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:45,504 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,504 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:45,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:45,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f758bc0[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:45,506 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:45,506 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,507 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,507 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:45,507 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:45,507 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:45,507 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,507 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:45,509 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:45,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:45,509 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:45,509 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:45,509 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:45,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:45,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:45,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:45,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:45,511 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:45,512 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:45,512 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:45,512 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:45,512 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:45,513 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,513 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,514 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:45,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:45,514 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:45,515 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,517 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,517 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,517 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,517 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,517 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,517 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:45,517 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:45,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:45,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:45,517 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,518 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:45,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:45,518 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:45,518 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:45,518 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:45,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:45,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:45,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36772 deadline: 1681844145607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:45,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36696 deadline: 1681844145609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,618 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:45,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:45,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:45,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:45,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:45,622 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:45,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:45,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:45,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:45,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,624 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:45,624 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:45,624 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,624 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:45,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:45,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:45,625 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:45,625 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:45,625 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:45,625 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:45,625 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:45,625 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:45,625 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:45,625 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:45,625 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:45,625 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:45,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:45,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:45,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:45,628 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,628 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:45,629 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,629 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:45,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:45,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:45,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:45,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:45,630 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:45,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@240b2867[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:45,630 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,631 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,631 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:45,631 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:45,631 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:45,631 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,632 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:45,633 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,633 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:45,633 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:45,633 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:45,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:45,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:45,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:45,635 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:45,635 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:45,635 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:45,635 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:45,635 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:45,637 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,637 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:45,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,637 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,637 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:45,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,638 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,642 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:45,642 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:45,642 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:45,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:45,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:45,642 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:45,642 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,642 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,642 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:45,642 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:45,642 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,642 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,642 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,642 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:45,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:45,707 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/21a631e04ad24281acefe3af5b71ce97 2023-04-18 18:54:45,727 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/95523899589f420fa2b42d157513d30a 2023-04-18 18:54:45,742 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:45,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:45,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:45,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:45,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:45,752 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:45,752 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:45,752 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:45,752 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:45,752 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,752 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,753 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/f9c75fe9139e4fef8f8ed687cb8eb917 2023-04-18 18:54:45,754 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:45,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:45,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:45,754 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,755 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,755 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:45,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:45,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:45,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:45,755 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:45,755 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:45,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:45,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:45,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:45,756 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:45,756 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:45,756 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:45,756 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:45,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:45,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:45,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:45,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:45,758 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,758 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:45,758 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/21a631e04ad24281acefe3af5b71ce97 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/21a631e04ad24281acefe3af5b71ce97 2023-04-18 18:54:45,758 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,758 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:45,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:45,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:45,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:45,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38ed284d[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:45,760 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:45,761 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,762 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,762 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:45,762 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:45,762 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:45,762 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,762 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:45,763 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/21a631e04ad24281acefe3af5b71ce97, entries=150, sequenceid=111, filesize=11.6 K 2023-04-18 18:54:45,764 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/95523899589f420fa2b42d157513d30a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/95523899589f420fa2b42d157513d30a 2023-04-18 18:54:45,768 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:45,768 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:45,768 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:45,769 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:45,769 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/95523899589f420fa2b42d157513d30a, entries=150, sequenceid=111, filesize=11.6 K 2023-04-18 18:54:45,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:45,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:45,770 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/f9c75fe9139e4fef8f8ed687cb8eb917 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/f9c75fe9139e4fef8f8ed687cb8eb917 2023-04-18 18:54:45,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:45,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:45,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:45,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:45,772 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:45,772 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:45,772 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:45,772 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:45,772 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:45,774 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/f9c75fe9139e4fef8f8ed687cb8eb917, entries=150, sequenceid=111, filesize=11.6 K 2023-04-18 18:54:45,775 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~436.03 KB/446496, currentSize=26.84 KB/27480 for e3c3c3367729f1159cdc97339f70376a in 479ms, sequenceid=111, compaction requested=true 2023-04-18 18:54:45,775 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:45,775 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:45,775 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:45,775 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:45,775 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:54:45,775 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:54:45,776 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:54:45,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,777 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/A is initiating minor compaction (all files) 2023-04-18 18:54:45,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:45,777 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,777 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/A in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:45,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,777 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/ded014cc35fa4880ada9724ed3b5158f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/4a295d84e69f4ba09f035db3fe13c51e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/5a52b1f63e0e4b27a96dca32ba575b9e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/21a631e04ad24281acefe3af5b71ce97] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=44.1 K 2023-04-18 18:54:45,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,777 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:54:45,777 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/B is initiating minor compaction (all files) 2023-04-18 18:54:45,777 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:45,777 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/B in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:45,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:45,777 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/60de32464a4545e6945720fb7fcfc843, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/0a230ea48e6b417db6923b93090721c8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/8c786d850e68420f9c80cb61b5a8b2ae, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/95523899589f420fa2b42d157513d30a] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=44.1 K 2023-04-18 18:54:45,777 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting ded014cc35fa4880ada9724ed3b5158f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681844081978 2023-04-18 18:54:45,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:45,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:45,778 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 60de32464a4545e6945720fb7fcfc843, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681844081978 2023-04-18 18:54:45,778 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 4a295d84e69f4ba09f035db3fe13c51e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681844082064 2023-04-18 18:54:45,778 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:45,778 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 0a230ea48e6b417db6923b93090721c8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681844082064 2023-04-18 18:54:45,779 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 5a52b1f63e0e4b27a96dca32ba575b9e, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1681844083170 2023-04-18 18:54:45,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,779 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 8c786d850e68420f9c80cb61b5a8b2ae, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1681844083170 2023-04-18 18:54:45,779 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 21a631e04ad24281acefe3af5b71ce97, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1681844084227 2023-04-18 18:54:45,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:45,779 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 95523899589f420fa2b42d157513d30a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1681844084227 2023-04-18 18:54:45,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,780 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:45,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:45,783 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:45,783 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:45,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:45,783 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:45,783 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:45,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:45,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:45,783 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,783 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,783 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,783 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,783 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:45,783 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:45,783 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,783 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:45,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:45,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:45,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:45,789 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#A#compaction#206 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:45,790 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#B#compaction#207 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:45,819 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/f7bb2dd03b364ccca404cccdf5af21b5 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/f7bb2dd03b364ccca404cccdf5af21b5 2023-04-18 18:54:45,825 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/B of e3c3c3367729f1159cdc97339f70376a into f7bb2dd03b364ccca404cccdf5af21b5(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:45,825 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:45,825 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/B, priority=12, startTime=1681844085775; duration=0sec 2023-04-18 18:54:45,826 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:45,826 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:54:45,827 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:54:45,827 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/C is initiating minor compaction (all files) 2023-04-18 18:54:45,827 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/C in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:45,827 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/6cae18f511504c9aa01e981e241d7186, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/fd89a62a7bfa4b758ef02e83f286009d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b0917fd341184707be47c165186394df, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/f9c75fe9139e4fef8f8ed687cb8eb917] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=44.1 K 2023-04-18 18:54:45,828 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 6cae18f511504c9aa01e981e241d7186, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681844081978 2023-04-18 18:54:45,828 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting fd89a62a7bfa4b758ef02e83f286009d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681844082064 2023-04-18 18:54:45,828 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b0917fd341184707be47c165186394df, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1681844083170 2023-04-18 18:54:45,829 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting f9c75fe9139e4fef8f8ed687cb8eb917, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1681844084227 2023-04-18 18:54:45,836 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#C#compaction#208 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:45,868 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/4b035617edec47b69a58cb13893f317c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/4b035617edec47b69a58cb13893f317c 2023-04-18 18:54:45,876 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/C of e3c3c3367729f1159cdc97339f70376a into 4b035617edec47b69a58cb13893f317c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:45,876 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:45,876 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/C, priority=12, startTime=1681844085775; duration=0sec 2023-04-18 18:54:45,876 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:45,884 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:45,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:45,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:45,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:45,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:45,901 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:45,901 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:45,901 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:45,901 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:45,902 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,902 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,903 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:45,903 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:45,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:45,903 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:45,904 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:45,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:45,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:45,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:45,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:45,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:45,905 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:45,905 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:45,905 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:45,905 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:45,905 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:45,905 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:45,905 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:45,905 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-18 18:54:45,905 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:45,905 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:45,905 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:45,906 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:45,906 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:45,906 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:45,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:45,914 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:45,915 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/5a08f7d605a34cb28e31d7eb44fe7d57 2023-04-18 18:54:45,925 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/1db7b88f1c2f42fbac872559507c391e 2023-04-18 18:54:45,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844085931, max timestamp=1681844085936Num uniques 859; 2023-04-18 18:54:45,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844085931, max timestamp=1681844085936Num uniques 859; 2023-04-18 18:54:45,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:54:45,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844085931, max timestamp=1681844085936Num uniques 859; 2023-04-18 18:54:45,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:54:45,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:54:45,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681844085955, max timestamp=1681844085964Num uniques 1018; 2023-04-18 18:54:45,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-18 18:54:45,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681844085955, max timestamp=1681844085964Num uniques 1018; 2023-04-18 18:54:45,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-18 18:54:45,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681844085955, max timestamp=1681844085964Num uniques 1018; 2023-04-18 18:54:45,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-18 18:54:46,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681844085989, max timestamp=1681844085998Num uniques 1077; 2023-04-18 18:54:46,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-18 18:54:46,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681844085989, max timestamp=1681844085998Num uniques 1077; 2023-04-18 18:54:46,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-18 18:54:46,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681844085989, max timestamp=1681844085998Num uniques 1077; 2023-04-18 18:54:46,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-18 18:54:46,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681844086050, max timestamp=1681844086057Num uniques 1086; 2023-04-18 18:54:46,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-18 18:54:46,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681844086050, max timestamp=1681844086057Num uniques 1086; 2023-04-18 18:54:46,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-18 18:54:46,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681844086050, max timestamp=1681844086057Num uniques 1086; 2023-04-18 18:54:46,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-18 18:54:46,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681844086078, max timestamp=1681844086083Num uniques 1045; 2023-04-18 18:54:46,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-18 18:54:46,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681844086078, max timestamp=1681844086083Num uniques 1045; 2023-04-18 18:54:46,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-18 18:54:46,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681844086078, max timestamp=1681844086083Num uniques 1045; 2023-04-18 18:54:46,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-18 18:54:46,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681844086106, max timestamp=1681844086114Num uniques 1054; 2023-04-18 18:54:46,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-18 18:54:46,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681844086106, max timestamp=1681844086114Num uniques 1054; 2023-04-18 18:54:46,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-18 18:54:46,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681844086106, max timestamp=1681844086114Num uniques 1054; 2023-04-18 18:54:46,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-18 18:54:46,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681844086134, max timestamp=1681844086149Num uniques 1063; 2023-04-18 18:54:46,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681844086134, max timestamp=1681844086149Num uniques 1063; 2023-04-18 18:54:46,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-18 18:54:46,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-18 18:54:46,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681844086134, max timestamp=1681844086149Num uniques 1063; 2023-04-18 18:54:46,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-18 18:54:46,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681844086168, max timestamp=1681844086178Num uniques 1022; 2023-04-18 18:54:46,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-18 18:54:46,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681844086168, max timestamp=1681844086178Num uniques 1022; 2023-04-18 18:54:46,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-18 18:54:46,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681844086168, max timestamp=1681844086178Num uniques 1022; 2023-04-18 18:54:46,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-18 18:54:46,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681844086199, max timestamp=1681844086208Num uniques 1031; 2023-04-18 18:54:46,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-18 18:54:46,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681844086199, max timestamp=1681844086208Num uniques 1031; 2023-04-18 18:54:46,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-18 18:54:46,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681844086199, max timestamp=1681844086208Num uniques 1031; 2023-04-18 18:54:46,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-18 18:54:46,214 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/f489bbb9a94e4bd68632003dd3941b3b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/f489bbb9a94e4bd68632003dd3941b3b 2023-04-18 18:54:46,219 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/A of e3c3c3367729f1159cdc97339f70376a into f489bbb9a94e4bd68632003dd3941b3b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:46,219 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:46,219 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/A, priority=12, startTime=1681844085775; duration=0sec 2023-04-18 18:54:46,219 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:46,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681844086230, max timestamp=1681844086237Num uniques 1040; 2023-04-18 18:54:46,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-18 18:54:46,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681844086230, max timestamp=1681844086237Num uniques 1040; 2023-04-18 18:54:46,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681844086230, max timestamp=1681844086237Num uniques 1040; 2023-04-18 18:54:46,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-18 18:54:46,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-18 18:54:46,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681844086258, max timestamp=1681844086264Num uniques 1099; 2023-04-18 18:54:46,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-18 18:54:46,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681844086240, max timestamp=1681844086264Num uniques 1099; 2023-04-18 18:54:46,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-18 18:54:46,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681844086258, max timestamp=1681844086264Num uniques 1099; 2023-04-18 18:54:46,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-18 18:54:46,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681844086288, max timestamp=1681844086293Num uniques 1108; 2023-04-18 18:54:46,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681844086288, max timestamp=1681844086293Num uniques 1058; 2023-04-18 18:54:46,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681844086288, max timestamp=1681844086293Num uniques 1058; 2023-04-18 18:54:46,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-18 18:54:46,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-18 18:54:46,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-18 18:54:46,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681844086313, max timestamp=1681844086319Num uniques 1017; 2023-04-18 18:54:46,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-18 18:54:46,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681844086313, max timestamp=1681844086319Num uniques 1017; 2023-04-18 18:54:46,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-18 18:54:46,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681844086313, max timestamp=1681844086319Num uniques 1017; 2023-04-18 18:54:46,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-18 18:54:46,343 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/5ae29c80b457462ab7ce0c66a5a1595a 2023-04-18 18:54:46,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681844086335, max timestamp=1681844086347Num uniques 1026; 2023-04-18 18:54:46,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681844086335, max timestamp=1681844086347Num uniques 1026; 2023-04-18 18:54:46,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-18 18:54:46,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-18 18:54:46,351 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/5a08f7d605a34cb28e31d7eb44fe7d57 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/5a08f7d605a34cb28e31d7eb44fe7d57 2023-04-18 18:54:46,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681844086335, max timestamp=1681844086347Num uniques 1026; 2023-04-18 18:54:46,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=6870, suffix heap size=134672,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=150 2023-04-18 18:54:46,362 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/5a08f7d605a34cb28e31d7eb44fe7d57, entries=150, sequenceid=120, filesize=11.6 K 2023-04-18 18:54:46,363 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/1db7b88f1c2f42fbac872559507c391e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/1db7b88f1c2f42fbac872559507c391e 2023-04-18 18:54:46,367 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/1db7b88f1c2f42fbac872559507c391e, entries=150, sequenceid=120, filesize=11.6 K 2023-04-18 18:54:46,368 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/5ae29c80b457462ab7ce0c66a5a1595a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/5ae29c80b457462ab7ce0c66a5a1595a 2023-04-18 18:54:46,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1681844086358, max timestamp=1681844086367Num uniques 1035; 2023-04-18 18:54:46,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-04-18 18:54:46,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1681844086358, max timestamp=1681844086367Num uniques 1035; 2023-04-18 18:54:46,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1681844086358, max timestamp=1681844086367Num uniques 1009; 2023-04-18 18:54:46,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-04-18 18:54:46,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=13050, suffix heap size=132904,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=285 2023-04-18 18:54:46,372 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/5ae29c80b457462ab7ce0c66a5a1595a, entries=150, sequenceid=120, filesize=11.6 K 2023-04-18 18:54:46,373 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=53.67 KB/54960 for e3c3c3367729f1159cdc97339f70376a in 467ms, sequenceid=120, compaction requested=false 2023-04-18 18:54:46,373 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:46,373 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:46,373 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:46,373 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:46,373 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,373 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:46,373 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:46,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:46,375 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=60.38 KB heapSize=154.03 KB 2023-04-18 18:54:46,376 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:46,376 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:46,376 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:46,376 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:46,376 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:46,376 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:46,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,380 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:46,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:46,380 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,380 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:46,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:46,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:46,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:46,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:46,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36b0cebb[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:46,382 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:46,382 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,383 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,384 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,384 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:46,384 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:46,384 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:46,384 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:46,385 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:46,385 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:46,386 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:46,386 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:46,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:46,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:46,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:46,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:46,390 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:46,390 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:46,390 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:46,390 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:46,390 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:46,392 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,392 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:46,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,392 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:46,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:46,393 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:46,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,394 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:46,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844146392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681844086387, max timestamp=1681844086396Num uniques 859; 2023-04-18 18:54:46,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 18:54:46,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681844086387, max timestamp=1681844086396Num uniques 859; 2023-04-18 18:54:46,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681844086387, max timestamp=1681844086396Num uniques 859; 2023-04-18 18:54:46,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 18:54:46,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 18:54:46,399 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:46,399 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:46,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:46,399 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:46,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:46,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:46,399 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:46,399 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,399 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,399 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:46,399 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,399 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:46,399 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:46,399 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:46,399 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:46,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844146408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:46,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844146411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:46,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844146412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:46,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844146412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:46,499 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:46,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844146498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:46,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:46,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:46,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:46,503 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:46,503 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:46,503 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:46,503 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:46,503 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,503 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,504 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:46,505 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:46,505 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,505 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,506 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:46,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:46,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:46,506 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:46,506 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:46,507 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:46,507 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:46,507 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:46,507 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:46,507 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,507 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:46,507 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:46,508 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,508 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:46,509 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,509 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:46,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:46,510 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:46,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45e86826[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:46,510 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,511 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,511 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:46,511 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:46,511 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,511 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:46,511 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:46,514 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,514 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:46,514 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:46,514 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:46,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:46,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844146513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:46,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:46,517 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:46,517 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:46,517 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:46,517 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:46,517 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:46,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:46,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844146515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,519 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,519 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:46,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,519 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:46,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:46,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:46,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844146517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:46,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:46,519 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:46,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:46,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844146517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,521 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,523 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,523 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:46,523 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:46,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:46,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:46,523 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:46,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:46,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:46,523 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,524 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:46,524 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,524 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:46,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:46,524 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,524 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,524 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,524 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:46,624 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:46,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:46,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:46,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:46,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:46,627 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:46,628 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:46,628 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:46,628 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:46,628 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,628 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,629 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:46,629 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:46,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:46,630 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,630 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:46,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:46,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,630 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:46,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:46,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:46,631 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:46,631 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:46,631 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:46,631 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:46,632 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:46,632 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,632 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:46,632 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:46,634 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,634 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:46,635 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,635 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:46,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:46,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@385d38ec[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:46,636 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:46,637 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,638 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,638 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:46,638 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:46,638 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:46,638 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,638 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:46,640 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,640 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:46,640 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:46,640 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:46,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:46,642 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:46,643 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:46,643 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:46,643 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:46,643 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:46,644 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,644 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,644 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:46,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:46,645 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:46,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:46,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,646 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,652 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,652 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:46,652 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:46,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:46,652 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:46,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:46,652 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:46,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:46,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:46,652 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:46,653 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,653 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:46,653 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:46,653 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,653 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,653 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:46,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844146700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:46,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844146715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:46,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844146720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:46,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844146721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:46,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844146721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,753 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:46,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:46,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:46,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:46,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:46,756 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:46,756 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:46,756 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:46,756 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:46,757 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,757 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,758 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,758 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:46,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:46,758 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,759 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:46,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:46,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:46,760 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:46,760 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:46,760 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:46,760 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:46,760 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:46,760 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:46,760 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:46,760 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:46,760 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:46,760 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:46,760 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:46,761 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,761 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:46,761 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:46,762 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,762 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:46,763 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,763 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:46,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:46,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6923c0ea[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:46,765 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:46,765 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,767 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,768 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:46,768 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:46,768 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,768 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:46,768 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:46,770 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:46,770 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:46,770 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:46,770 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:46,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:46,774 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:46,774 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:46,774 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:46,774 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:46,774 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:46,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:46,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:46,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:46,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:46,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:46,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,777 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,779 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,785 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,785 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,785 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,785 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,785 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:46,785 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:46,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:46,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:46,785 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:46,785 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:46,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:46,786 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:46,786 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:46,786 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,786 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:46,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:46,787 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/1318b43be7a2491ea10792c6f55ed019 2023-04-18 18:54:46,819 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/60b86b29438241869d747a4bced1021b 2023-04-18 18:54:46,886 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:46,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:46,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:46,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:46,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:46,891 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:46,891 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:46,891 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:46,891 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:46,892 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,892 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,895 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:46,895 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:46,895 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,895 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:46,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:46,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:46,897 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:46,897 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:46,897 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:46,897 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:46,897 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:46,897 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:46,897 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:46,897 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:46,898 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:46,898 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:46,898 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:46,898 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,898 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:46,898 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:46,899 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,899 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:46,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:46,900 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,900 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:46,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:46,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:46,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:46,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:46,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59561e63[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:46,902 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:46,902 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,903 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,903 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:46,903 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:46,903 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:46,903 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,903 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:46,905 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,905 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:46,905 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:46,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:46,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:46,906 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:46,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:46,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:46,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:46,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:46,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:46,908 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:46,909 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:46,909 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:46,909 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:46,909 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:46,912 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,913 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:46,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:46,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,913 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:46,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:46,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:46,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:46,913 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,915 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:46,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:46,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,918 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,918 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,918 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:46,918 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:46,918 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:46,918 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,918 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:46,918 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:46,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:46,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:46,918 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:46,918 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:46,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:46,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:46,918 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:46,918 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:46,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:47,005 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:47,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844147005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,018 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:47,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:47,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:47,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:47,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844147019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:47,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:47,023 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:47,023 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:47,023 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:47,023 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:47,023 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,023 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:47,025 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,025 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:47,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:47,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:47,026 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,026 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:47,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:47,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:47,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:47,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:47,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:47,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844147024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:47,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844147024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:47,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:47,027 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:47,027 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:47,027 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:47,028 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:47,028 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:47,028 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:47,028 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:47,028 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:47,028 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:47,028 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:47,028 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:47,028 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,028 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:47,028 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:47,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:47,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844147026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,030 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,030 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:47,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:47,030 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,030 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:47,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:47,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:47,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:47,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:47,032 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:47,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ff9f437[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:47,032 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,034 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,034 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:47,034 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:47,034 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:47,035 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,035 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:47,036 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:47,036 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:47,036 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,037 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:47,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:47,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:47,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:47,039 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:47,039 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:47,039 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:47,039 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:47,039 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:47,041 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:47,041 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,041 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:47,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:47,042 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,044 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,051 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,051 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:47,051 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:47,051 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,051 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,051 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:47,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:47,051 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:47,051 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:47,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:47,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:47,051 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:47,052 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:47,051 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:47,051 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:47,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:47,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:47,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:47,152 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:47,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:47,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:47,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:47,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:47,155 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:47,155 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:47,155 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:47,155 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:47,156 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,156 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:47,158 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:47,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:47,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:47,158 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:47,158 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,158 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:47,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:47,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:47,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:47,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:47,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:47,160 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:47,160 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:47,160 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:47,160 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:47,160 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:47,160 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:47,160 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:47,160 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:47,160 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:47,160 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:47,160 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:47,160 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,160 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:47,160 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:47,162 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,162 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:47,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:47,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:47,163 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,163 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:47,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:47,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:47,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:47,165 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:47,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e322d7c[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:47,165 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,166 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,166 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:47,166 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:47,166 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:47,166 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,166 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:47,169 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,169 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:47,169 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:47,169 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:47,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:47,171 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:47,171 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:47,171 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:47,171 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:47,171 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:47,173 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,173 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,173 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:47,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:47,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:47,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:47,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:47,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:47,174 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,175 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,178 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,178 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:47,178 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:47,178 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,178 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,178 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,178 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:47,178 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:47,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:47,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:47,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:47,178 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:47,178 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:47,178 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:47,178 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:47,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:47,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:47,233 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/bab22ccb86a44aff8781e5edef1d38f7 2023-04-18 18:54:47,238 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/1318b43be7a2491ea10792c6f55ed019 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/1318b43be7a2491ea10792c6f55ed019 2023-04-18 18:54:47,242 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/1318b43be7a2491ea10792c6f55ed019, entries=150, sequenceid=386, filesize=11.9 K 2023-04-18 18:54:47,243 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/60b86b29438241869d747a4bced1021b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/60b86b29438241869d747a4bced1021b 2023-04-18 18:54:47,249 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/60b86b29438241869d747a4bced1021b, entries=150, sequenceid=386, filesize=11.9 K 2023-04-18 18:54:47,250 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/bab22ccb86a44aff8781e5edef1d38f7 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bab22ccb86a44aff8781e5edef1d38f7 2023-04-18 18:54:47,253 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bab22ccb86a44aff8781e5edef1d38f7, entries=150, sequenceid=386, filesize=11.9 K 2023-04-18 18:54:47,254 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~173.91 KB/178080, currentSize=127.47 KB/130530 for e3c3c3367729f1159cdc97339f70376a in 879ms, sequenceid=386, compaction requested=true 2023-04-18 18:54:47,254 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:47,254 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:47,254 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:47,254 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:54:47,254 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:54:47,254 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:47,255 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36096 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:54:47,255 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36096 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:54:47,255 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/B is initiating minor compaction (all files) 2023-04-18 18:54:47,256 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/A is initiating minor compaction (all files) 2023-04-18 18:54:47,256 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/B in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:47,256 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/A in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:47,256 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/f7bb2dd03b364ccca404cccdf5af21b5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/1db7b88f1c2f42fbac872559507c391e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/60b86b29438241869d747a4bced1021b] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=35.3 K 2023-04-18 18:54:47,256 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/f489bbb9a94e4bd68632003dd3941b3b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/5a08f7d605a34cb28e31d7eb44fe7d57, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/1318b43be7a2491ea10792c6f55ed019] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=35.3 K 2023-04-18 18:54:47,256 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting f7bb2dd03b364ccca404cccdf5af21b5, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1681844084227 2023-04-18 18:54:47,256 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting f489bbb9a94e4bd68632003dd3941b3b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1681844084227 2023-04-18 18:54:47,257 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 1db7b88f1c2f42fbac872559507c391e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1681844085298 2023-04-18 18:54:47,257 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 5a08f7d605a34cb28e31d7eb44fe7d57, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1681844085298 2023-04-18 18:54:47,260 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 60b86b29438241869d747a4bced1021b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1681844086362 2023-04-18 18:54:47,260 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 1318b43be7a2491ea10792c6f55ed019, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1681844086362 2023-04-18 18:54:47,277 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#A#compaction#215 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:47,278 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:47,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:47,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:47,293 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#B#compaction#216 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:47,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:47,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:47,296 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:47,296 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:47,296 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:47,296 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:47,297 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,297 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:47,298 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:47,298 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:47,299 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,299 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:47,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:47,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:47,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:47,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:47,314 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:47,314 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:47,314 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:47,314 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:47,314 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=127.47 KB heapSize=351 KB 2023-04-18 18:54:47,315 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:47,315 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:47,315 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:47,315 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:47,315 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:47,315 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:47,322 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/9f7f9782511e492a95c0463ffc111d21 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/9f7f9782511e492a95c0463ffc111d21 2023-04-18 18:54:47,327 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/B of e3c3c3367729f1159cdc97339f70376a into 9f7f9782511e492a95c0463ffc111d21(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:47,327 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:47,327 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/B, priority=13, startTime=1681844087254; duration=0sec 2023-04-18 18:54:47,327 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:47,327 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:54:47,328 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36096 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:54:47,328 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/C is initiating minor compaction (all files) 2023-04-18 18:54:47,329 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/C in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:47,329 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/4b035617edec47b69a58cb13893f317c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/5ae29c80b457462ab7ce0c66a5a1595a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bab22ccb86a44aff8781e5edef1d38f7] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=35.3 K 2023-04-18 18:54:47,329 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 4b035617edec47b69a58cb13893f317c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1681844084227 2023-04-18 18:54:47,329 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 5ae29c80b457462ab7ce0c66a5a1595a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1681844085298 2023-04-18 18:54:47,330 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting bab22ccb86a44aff8781e5edef1d38f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1681844086362 2023-04-18 18:54:47,331 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/2ecb14c5e57c4b1e8c0fdb6a2c141895 2023-04-18 18:54:47,338 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#C#compaction#218 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:47,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:47,511 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:47,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:47,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844147529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:47,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844147530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:47,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844147530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:47,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844147531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:47,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844147535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:47,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844147634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:47,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844147634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,722 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/807a8db805954a9a915237af046ba6a8 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/807a8db805954a9a915237af046ba6a8 2023-04-18 18:54:47,728 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/A of e3c3c3367729f1159cdc97339f70376a into 807a8db805954a9a915237af046ba6a8(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:47,728 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:47,728 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/A, priority=13, startTime=1681844087254; duration=0sec 2023-04-18 18:54:47,728 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:47,759 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/d234e63ab9f64a3e828b9dc625246acf 2023-04-18 18:54:47,759 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/bc24fab781c946828b93b0710f5966a2 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bc24fab781c946828b93b0710f5966a2 2023-04-18 18:54:47,767 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/C of e3c3c3367729f1159cdc97339f70376a into bc24fab781c946828b93b0710f5966a2(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:47,767 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:47,767 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/C, priority=13, startTime=1681844087254; duration=0sec 2023-04-18 18:54:47,767 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:47,787 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/8d0eba14a5e347969fb0889173a736e2 2023-04-18 18:54:47,792 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/2ecb14c5e57c4b1e8c0fdb6a2c141895 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/2ecb14c5e57c4b1e8c0fdb6a2c141895 2023-04-18 18:54:47,799 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/2ecb14c5e57c4b1e8c0fdb6a2c141895, entries=150, sequenceid=419, filesize=11.9 K 2023-04-18 18:54:47,800 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/d234e63ab9f64a3e828b9dc625246acf as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/d234e63ab9f64a3e828b9dc625246acf 2023-04-18 18:54:47,804 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/d234e63ab9f64a3e828b9dc625246acf, entries=150, sequenceid=419, filesize=11.9 K 2023-04-18 18:54:47,805 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/8d0eba14a5e347969fb0889173a736e2 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/8d0eba14a5e347969fb0889173a736e2 2023-04-18 18:54:47,811 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/8d0eba14a5e347969fb0889173a736e2, entries=150, sequenceid=419, filesize=11.9 K 2023-04-18 18:54:47,811 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~350.95 KB/359376, currentSize=60.38 KB/61830 for e3c3c3367729f1159cdc97339f70376a in 497ms, sequenceid=419, compaction requested=false 2023-04-18 18:54:47,811 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:47,811 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:47,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:47,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:47,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:47,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:47,814 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,815 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:47,815 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,815 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:47,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:47,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@796188c0[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:47,817 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:47,817 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,819 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:47,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:47,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:47,819 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,819 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:47,820 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,820 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:47,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:47,825 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:47,825 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:47,825 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:47,825 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:47,825 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:47,827 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,827 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:47,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,828 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:47,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:47,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:47,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:47,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,828 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,829 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,833 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:47,833 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:47,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:47,833 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,833 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:47,833 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:47,834 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:47,834 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:47,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:47,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:47,834 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,834 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,834 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:47,834 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:47,834 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:47,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:47,841 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 18:54:47,842 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:47,842 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:47,842 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:47,842 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:47,842 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:47,842 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:47,856 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/fc4ac4a8606345a7807c569620a4e4d3 2023-04-18 18:54:47,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:47,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844147868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:47,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844147870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,934 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:47,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:47,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:47,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:47,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:47,937 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:47,937 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:47,938 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:47,938 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:47,938 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,938 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:47,939 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:47,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:47,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:47,939 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:47,940 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,940 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:47,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:47,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:47,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:47,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:47,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:47,941 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:47,941 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:47,941 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:47,941 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:47,941 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:47,941 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:47,941 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:47,942 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:47,942 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:47,942 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:47,942 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:47,942 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,942 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:47,942 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:47,943 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,943 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:47,944 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,944 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:47,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:47,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:47,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:47,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:47,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ad3489[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:47,945 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:47,946 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,947 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,947 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:47,947 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:47,947 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:47,947 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,947 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:47,950 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,950 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:47,950 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:47,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:47,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:47,950 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:47,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:47,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:47,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:47,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:47,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:47,952 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:47,952 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:47,952 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:47,952 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:47,952 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:47,954 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,954 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:47,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,954 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:47,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:47,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:47,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:47,955 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:47,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:47,956 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:47,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:47,958 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,958 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:47,958 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:47,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:47,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:47,958 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:47,959 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:47,959 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,959 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:47,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:47,959 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:47,959 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:47,959 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:47,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:47,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:47,959 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:47,959 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:47,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:47,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:47,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844147971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:47,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:47,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844147973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,060 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:48,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:48,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:48,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:48,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:48,064 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:48,064 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:48,064 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:48,064 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:48,066 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:48,066 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:48,067 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:48,067 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:48,068 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,068 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:48,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:48,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:48,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:48,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:48,072 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:48,072 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:48,072 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:48,073 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:48,073 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:48,073 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:48,073 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:48,073 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:48,073 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:48,073 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:48,073 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:48,073 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,073 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:48,073 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:48,081 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,081 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:48,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:48,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:48,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:48,082 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:48,082 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:48,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:48,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:48,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:48,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:48,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e532e7[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:48,083 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:48,083 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:48,084 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:48,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:48,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:48,084 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:48,084 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:48,084 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:48,085 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,085 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:48,086 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,086 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:48,086 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:48,086 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:48,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:48,089 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:48,089 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:48,089 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:48,089 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:48,089 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:48,091 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:48,091 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:48,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:48,091 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:48,091 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:48,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:48,093 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,096 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:48,096 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:48,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:48,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:48,096 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:48,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:48,096 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:48,096 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:48,096 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:48,096 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,096 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:48,097 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:48,097 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:48,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:48,097 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:48,097 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:48,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:48,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844148173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:48,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844148177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,197 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:48,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:48,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:48,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:48,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:48,201 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:48,201 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:48,201 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:48,201 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:48,201 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:48,201 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:48,202 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:48,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:48,202 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:48,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:48,203 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,203 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:48,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:48,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:48,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:48,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:48,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:48,204 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:48,204 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:48,204 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:48,204 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:48,204 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:48,204 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:48,204 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:48,205 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:48,205 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:48,205 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:48,205 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:48,205 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,205 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:48,205 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:48,206 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,206 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:48,207 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:48,207 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:48,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:48,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fa3b64a[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:48,208 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:48,209 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:48,210 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:48,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:48,210 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:48,210 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:48,210 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:48,210 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,210 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:48,213 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:48,213 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:48,213 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,213 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:48,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:48,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:48,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:48,216 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:48,216 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:48,216 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:48,216 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:48,216 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:48,218 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:48,218 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:48,218 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:48,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:48,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:48,219 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:48,220 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,223 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,223 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:48,223 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:48,223 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:48,223 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:48,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:48,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:48,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:48,223 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:48,223 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,224 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:48,224 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:48,224 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:48,224 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:48,224 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:48,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:48,269 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/79185b69e0d0469f880c2cc0ee48f973 2023-04-18 18:54:48,298 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/bdf24e6770cd4ddab49945acfc54b505 2023-04-18 18:54:48,303 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/fc4ac4a8606345a7807c569620a4e4d3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/fc4ac4a8606345a7807c569620a4e4d3 2023-04-18 18:54:48,307 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/fc4ac4a8606345a7807c569620a4e4d3, entries=200, sequenceid=436, filesize=14.3 K 2023-04-18 18:54:48,308 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/79185b69e0d0469f880c2cc0ee48f973 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/79185b69e0d0469f880c2cc0ee48f973 2023-04-18 18:54:48,316 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/79185b69e0d0469f880c2cc0ee48f973, entries=150, sequenceid=436, filesize=11.9 K 2023-04-18 18:54:48,316 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/bdf24e6770cd4ddab49945acfc54b505 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bdf24e6770cd4ddab49945acfc54b505 2023-04-18 18:54:48,321 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bdf24e6770cd4ddab49945acfc54b505, entries=150, sequenceid=436, filesize=11.9 K 2023-04-18 18:54:48,322 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for e3c3c3367729f1159cdc97339f70376a in 481ms, sequenceid=436, compaction requested=true 2023-04-18 18:54:48,322 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:48,322 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:48,322 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:54:48,322 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:54:48,322 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:48,323 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:48,324 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:48,324 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:54:48,324 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/A is initiating minor compaction (all files) 2023-04-18 18:54:48,324 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/A in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:48,324 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/807a8db805954a9a915237af046ba6a8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/2ecb14c5e57c4b1e8c0fdb6a2c141895, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/fc4ac4a8606345a7807c569620a4e4d3] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=38.3 K 2023-04-18 18:54:48,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:48,324 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36798 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:54:48,325 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/B is initiating minor compaction (all files) 2023-04-18 18:54:48,325 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/B in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:48,325 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/9f7f9782511e492a95c0463ffc111d21, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/d234e63ab9f64a3e828b9dc625246acf, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/79185b69e0d0469f880c2cc0ee48f973] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=35.9 K 2023-04-18 18:54:48,325 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 807a8db805954a9a915237af046ba6a8, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1681844086362 2023-04-18 18:54:48,326 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 9f7f9782511e492a95c0463ffc111d21, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1681844086362 2023-04-18 18:54:48,326 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting d234e63ab9f64a3e828b9dc625246acf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1681844086410 2023-04-18 18:54:48,326 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 2ecb14c5e57c4b1e8c0fdb6a2c141895, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1681844086410 2023-04-18 18:54:48,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:48,326 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 79185b69e0d0469f880c2cc0ee48f973, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1681844087526 2023-04-18 18:54:48,327 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting fc4ac4a8606345a7807c569620a4e4d3, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1681844087526 2023-04-18 18:54:48,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:48,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:48,328 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:48,329 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:48,329 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:48,329 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:48,329 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:48,329 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:48,331 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:48,331 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:48,331 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,331 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:48,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:48,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:48,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:48,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:48,334 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:48,334 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:48,334 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:48,336 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:48,336 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:48,336 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:48,336 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:48,337 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-18 18:54:48,337 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:48,337 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:48,337 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:48,337 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:48,337 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:48,337 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:48,341 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#A#compaction#224 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:48,341 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#B#compaction#225 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:48,371 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/d2234835849b44058ce2cd232d9d1eea as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/d2234835849b44058ce2cd232d9d1eea 2023-04-18 18:54:48,376 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/A of e3c3c3367729f1159cdc97339f70376a into d2234835849b44058ce2cd232d9d1eea(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:48,376 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:48,376 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/A, priority=13, startTime=1681844088322; duration=0sec 2023-04-18 18:54:48,376 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:48,376 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:54:48,377 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36798 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:54:48,377 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/C is initiating minor compaction (all files) 2023-04-18 18:54:48,377 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/C in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:48,377 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bc24fab781c946828b93b0710f5966a2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/8d0eba14a5e347969fb0889173a736e2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bdf24e6770cd4ddab49945acfc54b505] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=35.9 K 2023-04-18 18:54:48,378 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting bc24fab781c946828b93b0710f5966a2, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1681844086362 2023-04-18 18:54:48,378 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 8d0eba14a5e347969fb0889173a736e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1681844086410 2023-04-18 18:54:48,379 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting bdf24e6770cd4ddab49945acfc54b505, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1681844087526 2023-04-18 18:54:48,380 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/688920c1b0e34d9b96b7ba31335ec574 2023-04-18 18:54:48,404 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#C#compaction#228 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:48,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:48,477 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:48,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:48,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844148493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:48,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844148496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:48,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844148533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:48,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844148538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:48,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844148538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:48,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844148597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:48,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844148600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,759 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/53a8ec21aab24103b3997131abc7ff12 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/53a8ec21aab24103b3997131abc7ff12 2023-04-18 18:54:48,763 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/B of e3c3c3367729f1159cdc97339f70376a into 53a8ec21aab24103b3997131abc7ff12(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:48,764 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:48,764 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/B, priority=13, startTime=1681844088322; duration=0sec 2023-04-18 18:54:48,764 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:48,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:48,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844148799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:48,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844148803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:48,812 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/ea5e0d4c67054e538e5107a94f4f4fd4 2023-04-18 18:54:48,826 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/b7c00c149cd843cd9eb39deb4fc1c00e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b7c00c149cd843cd9eb39deb4fc1c00e 2023-04-18 18:54:48,830 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/C of e3c3c3367729f1159cdc97339f70376a into b7c00c149cd843cd9eb39deb4fc1c00e(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:48,831 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:48,831 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/C, priority=13, startTime=1681844088323; duration=0sec 2023-04-18 18:54:48,831 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:49,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:49,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:49,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844149103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844149104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,224 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/c5aac3f5231e4248bb91b9619025bc75 2023-04-18 18:54:49,228 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/688920c1b0e34d9b96b7ba31335ec574 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/688920c1b0e34d9b96b7ba31335ec574 2023-04-18 18:54:49,251 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/688920c1b0e34d9b96b7ba31335ec574, entries=150, sequenceid=454, filesize=11.9 K 2023-04-18 18:54:49,252 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/ea5e0d4c67054e538e5107a94f4f4fd4 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/ea5e0d4c67054e538e5107a94f4f4fd4 2023-04-18 18:54:49,267 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/ea5e0d4c67054e538e5107a94f4f4fd4, entries=150, sequenceid=454, filesize=11.9 K 2023-04-18 18:54:49,268 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/c5aac3f5231e4248bb91b9619025bc75 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/c5aac3f5231e4248bb91b9619025bc75 2023-04-18 18:54:49,272 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/c5aac3f5231e4248bb91b9619025bc75, entries=150, sequenceid=454, filesize=11.9 K 2023-04-18 18:54:49,273 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for e3c3c3367729f1159cdc97339f70376a in 936ms, sequenceid=454, compaction requested=false 2023-04-18 18:54:49,273 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:49,273 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:49,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:49,278 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,278 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:49,279 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:49,279 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:49,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:49,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a710af6[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:49,282 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:49,282 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:49,284 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:49,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:49,284 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:49,284 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:49,284 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:49,284 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,284 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:49,285 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:49,286 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:49,286 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:49,286 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:49,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:49,290 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:49,290 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:49,290 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:49,290 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:49,290 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:49,292 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:49,292 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:49,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:49,292 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:49,293 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:49,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:49,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:49,294 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:49,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:49,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:49,298 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:49,298 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:49,298 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:49,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:49,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:49,299 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:49,298 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:49,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:49,299 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:49,299 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:49,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:49,299 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,299 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:49,299 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:49,299 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:49,299 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:49,399 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:49,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:49,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:49,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:49,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:49,403 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:49,403 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:49,404 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:49,404 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:49,404 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:49,404 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:49,406 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:49,406 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:49,406 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:49,407 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:49,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:49,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:49,407 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:49,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:49,408 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:49,408 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:49,408 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:49,410 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:49,410 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:49,410 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:49,410 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:49,410 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-18 18:54:49,411 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:49,411 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:49,411 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:49,411 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:49,411 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:49,411 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:49,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:49,609 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:49,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:49,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844149631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:49,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844149633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:49,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844149734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:49,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844149736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,824 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/5568ef5003844af8b48ba0dc5c979ba8 2023-04-18 18:54:49,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:49,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844149938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:49,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:49,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844149941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,236 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/c564c2fe4eed43e8a5ffd80fc22b37dc 2023-04-18 18:54:50,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:50,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844150241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,247 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/0c9dfb27713d437f8494e5a4f1937880 2023-04-18 18:54:50,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:50,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844150246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,252 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/5568ef5003844af8b48ba0dc5c979ba8 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/5568ef5003844af8b48ba0dc5c979ba8 2023-04-18 18:54:50,259 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/5568ef5003844af8b48ba0dc5c979ba8, entries=150, sequenceid=471, filesize=11.9 K 2023-04-18 18:54:50,260 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/c564c2fe4eed43e8a5ffd80fc22b37dc as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/c564c2fe4eed43e8a5ffd80fc22b37dc 2023-04-18 18:54:50,264 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/c564c2fe4eed43e8a5ffd80fc22b37dc, entries=150, sequenceid=471, filesize=11.9 K 2023-04-18 18:54:50,265 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/0c9dfb27713d437f8494e5a4f1937880 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/0c9dfb27713d437f8494e5a4f1937880 2023-04-18 18:54:50,268 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/0c9dfb27713d437f8494e5a4f1937880, entries=150, sequenceid=471, filesize=11.9 K 2023-04-18 18:54:50,269 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for e3c3c3367729f1159cdc97339f70376a in 859ms, sequenceid=471, compaction requested=true 2023-04-18 18:54:50,269 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:50,269 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:50,269 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:50,270 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:50,270 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,270 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:50,270 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:50,271 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,271 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:50,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:50,272 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,272 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:50,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:50,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:50,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:50,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:50,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1381724d[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:50,275 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:50,275 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,284 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:50,285 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,285 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:50,288 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:50,288 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:50,288 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:50,288 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:50,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:50,293 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:50,293 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:50,293 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:50,293 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:50,293 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:50,295 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,295 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:50,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,295 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:50,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,297 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,299 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,301 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,301 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,301 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,301 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,301 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:50,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:50,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:50,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:50,301 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,301 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,301 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:50,301 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:50,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:50,301 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,301 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:50,302 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:50,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:50,402 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:50,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:50,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:50,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:50,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:50,405 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:50,405 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:50,405 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:50,405 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:50,406 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,406 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,407 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,407 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:50,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:50,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:50,407 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,408 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:50,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:50,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:50,409 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:50,409 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:50,409 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:50,409 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:50,409 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:50,409 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:50,410 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:50,410 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-18 18:54:50,410 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:50,410 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:50,410 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:50,410 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:50,410 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:50,410 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:50,418 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/76cf7abf00534c4e87ead50f1ec4ca49 2023-04-18 18:54:50,427 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/564b14ca32f149b5b21e61d216ecd781 2023-04-18 18:54:50,436 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/4ba2c3f7c3d74f9bb9a5c0f08091acb7 2023-04-18 18:54:50,439 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/76cf7abf00534c4e87ead50f1ec4ca49 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/76cf7abf00534c4e87ead50f1ec4ca49 2023-04-18 18:54:50,443 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/76cf7abf00534c4e87ead50f1ec4ca49, entries=150, sequenceid=489, filesize=11.9 K 2023-04-18 18:54:50,448 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/564b14ca32f149b5b21e61d216ecd781 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/564b14ca32f149b5b21e61d216ecd781 2023-04-18 18:54:50,458 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/564b14ca32f149b5b21e61d216ecd781, entries=150, sequenceid=489, filesize=11.9 K 2023-04-18 18:54:50,459 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/4ba2c3f7c3d74f9bb9a5c0f08091acb7 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/4ba2c3f7c3d74f9bb9a5c0f08091acb7 2023-04-18 18:54:50,463 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/4ba2c3f7c3d74f9bb9a5c0f08091acb7, entries=150, sequenceid=489, filesize=11.9 K 2023-04-18 18:54:50,464 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for e3c3c3367729f1159cdc97339f70376a in 54ms, sequenceid=489, compaction requested=true 2023-04-18 18:54:50,464 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:50,464 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:50,464 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:50,464 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:50,464 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,464 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:50,464 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:50,467 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,467 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:50,467 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,467 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:50,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:50,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a5a18d0[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:50,469 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:50,469 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,470 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,471 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:50,471 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:50,471 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:50,471 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,471 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:50,473 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,473 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:50,473 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:50,473 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:50,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:50,476 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:50,476 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:50,476 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:50,476 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:50,476 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:50,478 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,478 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:50,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,478 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:50,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:50,479 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,480 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,484 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,484 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:50,484 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:50,484 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,484 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,484 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,484 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:50,484 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:50,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:50,484 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:50,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:50,484 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:50,484 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:50,484 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:50,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:50,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:50,563 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 18:54:50,563 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:50,563 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:50,563 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:50,563 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:50,563 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:50,563 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:50,585 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:50,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681844090575, max timestamp=1681844090584Num uniques 859; 2023-04-18 18:54:50,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681844090575, max timestamp=1681844090584Num uniques 859; 2023-04-18 18:54:50,587 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 18:54:50,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:50,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681844090575, max timestamp=1681844090584Num uniques 859; 2023-04-18 18:54:50,587 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 18:54:50,587 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 18:54:50,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:50,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:50,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:50,589 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:50,589 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:50,590 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:50,590 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:50,590 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,590 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,591 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:50,591 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:50,592 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,592 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:50,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:50,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:50,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:50,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:50,593 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:50,593 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:50,593 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:50,594 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:50,594 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:50,594 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:50,594 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:50,596 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,596 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:50,596 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,596 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:50,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:50,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c9e837[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:50,598 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:50,598 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,599 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,599 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:50,599 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:50,599 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,599 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:50,599 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:50,602 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:50,602 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,602 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:50,602 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:50,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:50,604 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:50,604 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:50,604 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:50,604 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:50,604 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:50,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:50,606 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844150602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:50,606 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,606 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:50,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:50,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:50,607 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,608 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:50,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844150606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:50,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844150606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,610 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:50,610 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,610 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:50,611 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:50,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:50,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:50,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:50,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:50,611 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,611 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,611 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,611 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,611 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:50,611 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:50,611 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,611 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:50,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:50,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844150707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,711 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:50,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:50,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:50,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844150711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:50,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844150711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:50,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:50,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:50,715 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:50,715 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:50,715 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:50,715 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:50,715 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,715 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,716 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:50,716 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:50,717 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,717 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:50,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:50,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:50,718 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:50,718 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:50,718 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:50,718 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:50,718 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:50,718 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:50,718 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:50,719 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:50,719 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:50,719 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:50,719 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:50,719 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,719 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:50,719 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:50,721 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,721 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:50,722 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,722 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:50,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:50,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a0d6362[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:50,723 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:50,723 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,725 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,725 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:50,725 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:50,725 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,725 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:50,725 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:50,734 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:50,734 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:50,734 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,734 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:50,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:50,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:50,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:50,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:50,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:50,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:50,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:50,737 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:50,737 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:50,737 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:50,737 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:50,737 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:50,739 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,739 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,739 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:50,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:50,740 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,741 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,745 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,745 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:50,745 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,745 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:50,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:50,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:50,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:50,745 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:50,745 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,745 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,745 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,745 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,745 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,745 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:50,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:50,745 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:50,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:50,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:50,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844150747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:50,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844150749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,845 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:50,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:50,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:50,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:50,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:50,849 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:50,849 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:50,849 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:50,849 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:50,849 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,850 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,852 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,852 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:50,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:50,852 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,852 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,853 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:50,853 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:50,853 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:50,853 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:50,853 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:50,853 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:50,854 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:50,854 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:50,854 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:50,854 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:50,854 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:50,854 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:50,854 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:50,854 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:50,854 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,854 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:50,854 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:50,856 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:50,856 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:50,856 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,856 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:50,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:50,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42866174[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:50,858 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:50,858 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,861 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,861 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:50,861 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:50,861 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,861 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:50,861 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:50,862 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,862 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:50,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:50,862 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:50,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:50,862 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:50,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:50,865 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:50,865 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:50,865 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:50,865 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:50,865 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:50,866 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,867 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:50,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,867 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:50,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:50,867 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:50,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,868 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:50,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:50,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:50,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:50,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:50,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:50,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:50,871 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:50,871 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:50,871 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:50,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:50,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:50,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844150910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:50,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844150914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:50,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844150915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,971 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/2ff56496abca4374bc313d04c42365e1 2023-04-18 18:54:50,971 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:50,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:50,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:50,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:50,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:50,976 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:50,976 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:50,976 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:50,976 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:50,976 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,976 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,979 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:50,979 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:50,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:50,980 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,980 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:50,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:50,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:50,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:50,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:50,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:50,981 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:50,981 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:50,981 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:50,981 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:50,981 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:50,981 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:50,981 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:50,981 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:50,981 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:50,981 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:50,981 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:50,981 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,981 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:50,981 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:50,983 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,983 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:50,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:50,983 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,983 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:50,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:50,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:50,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28f07d6c[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:50,991 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:50,991 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,992 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:50,992 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:50,992 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:50,992 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:50,992 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,993 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:50,994 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:50,994 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,994 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:50,994 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:50,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:50,996 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:50,996 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:50,996 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:50,996 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:50,996 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:50,999 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,999 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:50,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,999 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:50,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:51,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:51,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:51,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:51,000 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,002 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,004 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:51,005 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:51,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:51,005 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,005 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:51,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:51,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:51,005 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:51,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:51,005 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,005 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:51,006 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:51,006 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,006 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,006 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,006 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,010 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/576bacdabcbe43ad858a6733b962f61f 2023-04-18 18:54:51,106 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:51,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:51,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:51,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:51,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:51,109 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:51,109 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:51,109 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:51,109 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:51,110 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,110 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,111 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:51,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:51,111 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,112 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,112 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:51,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:51,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:51,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:51,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:51,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:51,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:51,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:51,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:51,113 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:51,113 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:51,114 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:51,114 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:51,114 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:51,114 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:51,114 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,114 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:51,114 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:51,115 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,115 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:51,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:51,116 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,116 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:51,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:51,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f8fa015[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:51,117 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:51,117 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,119 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,119 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:51,119 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:51,119 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:51,119 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,119 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:51,120 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,120 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:51,120 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:51,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:51,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:51,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:51,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:51,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:51,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:51,123 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:51,123 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:51,123 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:51,123 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:51,123 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:51,125 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,125 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:51,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:51,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,125 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:51,125 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:51,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:51,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:51,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:51,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,126 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:51,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,132 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,132 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:51,132 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:51,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:51,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:51,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:51,132 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:51,132 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:51,133 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:51,133 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:51,132 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,133 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,133 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,133 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,133 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:51,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:51,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844151215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:51,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844151219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:51,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844151221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,233 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:51,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:51,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:51,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:51,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:51,238 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:51,238 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:51,238 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:51,238 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:51,239 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,239 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,240 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,240 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:51,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:51,240 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,240 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:51,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:51,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:51,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:51,241 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:51,242 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:51,242 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:51,242 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:51,242 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:51,242 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:51,242 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:51,242 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:51,242 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:51,242 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:51,242 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:51,242 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,242 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:51,242 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:51,244 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,244 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:51,244 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,244 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:51,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:51,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ea8def5[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:51,246 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:51,246 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,247 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,247 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:51,247 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:51,247 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,247 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:51,247 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:51,251 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:51,251 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:51,251 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:51,251 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:51,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:51,253 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:51,253 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:51,253 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:51,253 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:51,253 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:51,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:51,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:51,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:51,256 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,257 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,259 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,259 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,259 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,259 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,259 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,259 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:51,259 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,260 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:51,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:51,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:51,260 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:51,260 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:51,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:51,260 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:51,260 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:51,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:51,360 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:51,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:51,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:51,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:51,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:51,364 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:51,364 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:51,364 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:51,364 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:51,365 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,365 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,366 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,366 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:51,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:51,366 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,366 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:51,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:51,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:51,368 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:51,368 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:51,368 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:51,368 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:51,368 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:51,368 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:51,368 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:51,368 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:51,368 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,368 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:51,368 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:51,370 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,370 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:51,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:51,371 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,371 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:51,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:51,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:51,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:51,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:51,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fd48f95[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:51,372 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:51,372 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,374 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,374 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:51,374 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:51,374 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,374 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:51,374 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:51,375 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:51,375 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:51,376 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:51,376 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:51,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:51,378 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:51,378 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:51,378 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:51,378 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:51,378 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:51,379 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,379 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:51,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:51,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,380 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,381 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,385 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:51,385 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,385 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:51,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:51,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:51,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:51,385 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:51,385 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,385 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,385 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,385 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:51,385 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,386 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:51,386 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:51,386 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:51,428 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/b355a132009f4d199985a25eb4b6abf1 2023-04-18 18:54:51,432 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/2ff56496abca4374bc313d04c42365e1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/2ff56496abca4374bc313d04c42365e1 2023-04-18 18:54:51,436 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/2ff56496abca4374bc313d04c42365e1, entries=200, sequenceid=499, filesize=14.3 K 2023-04-18 18:54:51,437 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/576bacdabcbe43ad858a6733b962f61f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/576bacdabcbe43ad858a6733b962f61f 2023-04-18 18:54:51,440 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/576bacdabcbe43ad858a6733b962f61f, entries=150, sequenceid=499, filesize=11.9 K 2023-04-18 18:54:51,441 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/b355a132009f4d199985a25eb4b6abf1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b355a132009f4d199985a25eb4b6abf1 2023-04-18 18:54:51,445 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b355a132009f4d199985a25eb4b6abf1, entries=150, sequenceid=499, filesize=11.9 K 2023-04-18 18:54:51,445 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e3c3c3367729f1159cdc97339f70376a in 882ms, sequenceid=499, compaction requested=true 2023-04-18 18:54:51,446 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:51,446 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:51,446 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:54:51,446 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:54:51,446 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:51,446 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:51,447 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 63712 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:54:51,447 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/A is initiating minor compaction (all files) 2023-04-18 18:54:51,447 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/A in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:51,448 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61272 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:54:51,448 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/d2234835849b44058ce2cd232d9d1eea, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/688920c1b0e34d9b96b7ba31335ec574, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/5568ef5003844af8b48ba0dc5c979ba8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/76cf7abf00534c4e87ead50f1ec4ca49, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/2ff56496abca4374bc313d04c42365e1] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=62.2 K 2023-04-18 18:54:51,448 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/B is initiating minor compaction (all files) 2023-04-18 18:54:51,448 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting d2234835849b44058ce2cd232d9d1eea, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1681844087526 2023-04-18 18:54:51,448 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/B in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:51,448 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/53a8ec21aab24103b3997131abc7ff12, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/ea5e0d4c67054e538e5107a94f4f4fd4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/c564c2fe4eed43e8a5ffd80fc22b37dc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/564b14ca32f149b5b21e61d216ecd781, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/576bacdabcbe43ad858a6733b962f61f] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=59.8 K 2023-04-18 18:54:51,448 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 688920c1b0e34d9b96b7ba31335ec574, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681844087862 2023-04-18 18:54:51,449 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 53a8ec21aab24103b3997131abc7ff12, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1681844087526 2023-04-18 18:54:51,449 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 5568ef5003844af8b48ba0dc5c979ba8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1681844088487 2023-04-18 18:54:51,449 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting ea5e0d4c67054e538e5107a94f4f4fd4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681844087862 2023-04-18 18:54:51,449 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting c564c2fe4eed43e8a5ffd80fc22b37dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1681844088487 2023-04-18 18:54:51,449 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 76cf7abf00534c4e87ead50f1ec4ca49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1681844089621 2023-04-18 18:54:51,450 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 564b14ca32f149b5b21e61d216ecd781, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1681844089621 2023-04-18 18:54:51,450 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 2ff56496abca4374bc313d04c42365e1, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1681844090558 2023-04-18 18:54:51,450 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 576bacdabcbe43ad858a6733b962f61f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1681844090558 2023-04-18 18:54:51,458 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#B#compaction#240 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:51,459 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#A#compaction#239 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:51,474 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/b8a46de2595a4b0f89e97c3d82d11cb1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/b8a46de2595a4b0f89e97c3d82d11cb1 2023-04-18 18:54:51,478 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/B of e3c3c3367729f1159cdc97339f70376a into b8a46de2595a4b0f89e97c3d82d11cb1(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:51,478 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:51,478 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/B, priority=11, startTime=1681844091446; duration=0sec 2023-04-18 18:54:51,478 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:51,478 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:54:51,479 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61272 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:54:51,480 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/C is initiating minor compaction (all files) 2023-04-18 18:54:51,480 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/C in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:51,480 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b7c00c149cd843cd9eb39deb4fc1c00e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/c5aac3f5231e4248bb91b9619025bc75, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/0c9dfb27713d437f8494e5a4f1937880, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/4ba2c3f7c3d74f9bb9a5c0f08091acb7, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b355a132009f4d199985a25eb4b6abf1] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=59.8 K 2023-04-18 18:54:51,480 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b7c00c149cd843cd9eb39deb4fc1c00e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1681844087526 2023-04-18 18:54:51,480 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting c5aac3f5231e4248bb91b9619025bc75, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681844087862 2023-04-18 18:54:51,481 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 0c9dfb27713d437f8494e5a4f1937880, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1681844088487 2023-04-18 18:54:51,481 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 4ba2c3f7c3d74f9bb9a5c0f08091acb7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1681844089621 2023-04-18 18:54:51,481 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b355a132009f4d199985a25eb4b6abf1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1681844090558 2023-04-18 18:54:51,486 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:51,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:51,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:51,488 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#C#compaction#241 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:51,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:51,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:51,489 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:51,489 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:51,489 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:51,489 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:51,490 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,490 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,491 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:51,491 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:51,492 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,492 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:51,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:51,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:51,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:51,493 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:51,493 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:51,493 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-18 18:54:51,494 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:51,494 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:51,494 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:51,494 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:51,494 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:51,494 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:51,499 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/bebd2dd5b10c4498a3f9763b643289c2 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bebd2dd5b10c4498a3f9763b643289c2 2023-04-18 18:54:51,504 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/C of e3c3c3367729f1159cdc97339f70376a into bebd2dd5b10c4498a3f9763b643289c2(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:51,504 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:51,504 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/C, priority=11, startTime=1681844091446; duration=0sec 2023-04-18 18:54:51,504 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:51,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:51,724 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:51,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:51,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844151730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:51,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844151731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:51,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844151732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:51,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36688 deadline: 1681844151753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:51,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844151753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:51,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844151834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:51,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844151834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:51,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844151835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,874 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/3a1caf5c45e8436aaf444a09ede78e36 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/3a1caf5c45e8436aaf444a09ede78e36 2023-04-18 18:54:51,886 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/A of e3c3c3367729f1159cdc97339f70376a into 3a1caf5c45e8436aaf444a09ede78e36(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:51,886 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:51,886 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/A, priority=11, startTime=1681844091446; duration=0sec 2023-04-18 18:54:51,886 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:51,903 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/0c059abb2b964d91a6540ca35f09667e 2023-04-18 18:54:51,915 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/788cfaf5966b432b89987015941589cb 2023-04-18 18:54:51,926 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/2a14f50354f741c1b59bfb04a8e02558 2023-04-18 18:54:51,930 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/0c059abb2b964d91a6540ca35f09667e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/0c059abb2b964d91a6540ca35f09667e 2023-04-18 18:54:51,935 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/0c059abb2b964d91a6540ca35f09667e, entries=150, sequenceid=535, filesize=11.9 K 2023-04-18 18:54:51,935 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/788cfaf5966b432b89987015941589cb as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/788cfaf5966b432b89987015941589cb 2023-04-18 18:54:51,939 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/788cfaf5966b432b89987015941589cb, entries=150, sequenceid=535, filesize=11.9 K 2023-04-18 18:54:51,940 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/2a14f50354f741c1b59bfb04a8e02558 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/2a14f50354f741c1b59bfb04a8e02558 2023-04-18 18:54:51,944 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/2a14f50354f741c1b59bfb04a8e02558, entries=150, sequenceid=535, filesize=11.9 K 2023-04-18 18:54:51,945 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for e3c3c3367729f1159cdc97339f70376a in 452ms, sequenceid=535, compaction requested=false 2023-04-18 18:54:51,945 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:51,945 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:51,945 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:51,945 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:51,945 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,945 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:51,945 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:51,947 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:51,947 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:51,948 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,948 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:51,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:51,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34648a1e[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:51,950 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:51,950 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,951 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,951 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:51,951 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:51,951 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:51,951 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,951 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:51,954 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:51,954 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:51,954 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,954 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:51,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:51,957 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:51,957 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:51,957 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:51,957 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:51,957 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:51,959 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,959 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:51,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,959 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:51,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:51,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,961 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,961 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:51,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:51,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:51,971 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,971 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,971 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:51,971 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:51,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:51,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:51,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:51,971 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:51,971 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:51,971 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:51,971 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:51,972 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:51,972 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,972 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:51,972 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:51,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:51,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:52,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:52,040 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-18 18:54:52,040 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:52,040 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:52,040 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:52,040 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:52,040 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:52,040 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:52,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:52,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844152059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844092054, max timestamp=1681844092060Num uniques 859; 2023-04-18 18:54:52,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:54:52,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844092054, max timestamp=1681844092060Num uniques 859; 2023-04-18 18:54:52,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844092054, max timestamp=1681844092060Num uniques 859; 2023-04-18 18:54:52,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:54:52,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:54:52,072 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:52,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:52,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:52,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:52,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:52,076 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:52,076 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:52,076 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:52,076 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:52,076 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,076 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,079 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:52,079 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:52,079 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,079 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:52,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:52,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:52,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:52,080 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:52,080 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:52,080 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:52,081 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:52,081 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:52,081 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:52,081 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:52,081 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:52,081 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:52,081 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:52,081 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:52,081 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,081 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:52,081 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:52,082 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,083 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,083 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,083 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:52,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844152081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:52,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16491b8f[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:52,085 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:52,085 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,086 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:52,086 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,086 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:52,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:52,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844152084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,087 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,087 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,087 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:52,088 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:52,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:52,090 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:52,090 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:52,090 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:52,090 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:52,090 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:52,093 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,093 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:52,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,093 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,094 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,095 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,097 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,097 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,097 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,097 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,097 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:52,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:52,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:52,097 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,097 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:52,097 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,097 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:52,097 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:52,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:52,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:52,098 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:52,098 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:52,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:52,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:52,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:52,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844152163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:52,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844152185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:52,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844152187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,198 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:52,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:52,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:52,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:52,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:52,201 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:52,201 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:52,201 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:52,201 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:52,202 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,202 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,203 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,203 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:52,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:52,203 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,203 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:52,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:52,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:52,204 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:52,204 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:52,204 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:52,205 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:52,205 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:52,205 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:52,205 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:52,205 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:52,205 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:52,205 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:52,205 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:52,205 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,205 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:52,205 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:52,206 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,206 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,207 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,207 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:52,208 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:52,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ebe30c5[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:52,208 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,210 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,211 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,211 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:52,211 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:52,211 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:52,211 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:52,212 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:52,212 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:52,212 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,212 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:52,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:52,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:52,215 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:52,215 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:52,215 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:52,215 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:52,215 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:52,216 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,216 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,216 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:52,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,217 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,218 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,220 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:52,220 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,220 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:52,221 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:52,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:52,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:52,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:52,220 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,221 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:52,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:52,221 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:52,221 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:52,221 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:52,221 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,221 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,221 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,321 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:52,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:52,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:52,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:52,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:52,325 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:52,325 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:52,325 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:52,325 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:52,325 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,325 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,328 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,328 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:52,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:52,329 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,329 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:52,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:52,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:52,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:52,330 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:52,330 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:52,331 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:52,331 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:52,331 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:52,331 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:52,331 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:52,331 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:52,331 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:52,331 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:52,331 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:52,331 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,331 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:52,331 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:52,332 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,332 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,333 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,333 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:52,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ada8a0[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:52,334 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:52,334 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,335 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,336 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:52,336 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:52,336 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:52,336 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,336 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:52,337 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,337 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:52,337 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:52,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,337 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:52,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:52,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:52,340 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:52,340 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:52,340 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:52,340 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:52,340 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:52,342 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,342 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,343 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:52,343 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,344 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,347 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,347 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:52,347 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:52,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:52,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:52,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:52,347 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,347 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:52,347 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:52,347 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:52,347 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:52,347 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:52,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:52,348 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,348 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,348 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:52,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844152366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:52,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844152388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:52,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844152391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,447 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:52,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:52,448 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=549 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/936ad3a4f9d24b4fa524b19b3411cb31 2023-04-18 18:54:52,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:52,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:52,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:52,454 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:52,454 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:52,454 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:52,454 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:52,454 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,454 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,467 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:52,467 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:52,468 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,468 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:52,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:52,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:52,470 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:52,470 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:52,470 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:52,470 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:52,470 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:52,470 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:52,470 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:52,470 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:52,470 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:52,470 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:52,470 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:52,470 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,470 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:52,471 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:52,473 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,473 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,473 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,473 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:52,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1509e671[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:52,475 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:52,475 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,476 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,476 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:52,476 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:52,476 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:52,476 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,476 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:52,477 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,477 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:52,477 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:52,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,477 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:52,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:52,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:52,480 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:52,480 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:52,480 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:52,480 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:52,480 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:52,481 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,482 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,482 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:52,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:52,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:52,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,482 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,483 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,487 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,487 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:52,487 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:52,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:52,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:52,487 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:52,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:52,488 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:52,488 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,488 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:52,488 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:52,488 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:52,488 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,488 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,488 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,588 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:52,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:52,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:52,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:52,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:52,592 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:52,592 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:52,592 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:52,592 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:52,592 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,592 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,593 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,593 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:52,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:52,594 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,594 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:52,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:52,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:52,595 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:52,595 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:52,595 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:52,595 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:52,595 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:52,595 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:52,595 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:52,595 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:52,595 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:52,595 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:52,596 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:52,596 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,596 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:52,596 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:52,597 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,597 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,597 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,597 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:52,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bc2023c[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:52,599 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:52,599 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,600 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:52,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:52,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:52,600 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,600 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:52,602 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,603 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:52,603 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:52,603 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:52,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:52,605 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:52,605 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:52,605 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:52,605 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:52,605 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:52,606 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,606 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,606 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,607 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,608 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,611 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:52,611 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:52,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:52,611 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:52,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:52,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:52,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:52,611 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:52,611 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,612 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,612 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,612 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,612 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,612 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,612 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:52,612 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:52,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:52,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:52,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844152670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:52,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844152692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:52,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844152694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,712 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:52,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:52,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:52,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:52,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:52,716 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:52,716 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:52,716 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:52,716 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:52,717 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,717 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,718 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:52,718 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,718 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,719 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:52,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:52,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:52,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:52,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:52,720 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:52,720 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:52,720 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:52,720 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:52,720 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:52,720 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:52,720 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:52,720 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:52,720 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:52,720 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:52,720 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:52,720 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,720 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:52,720 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:52,723 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,723 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,724 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,724 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:52,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:52,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b53b99[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:52,726 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:52,726 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,727 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,727 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:52,727 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,727 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:52,727 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:52,727 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:52,729 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:52,729 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,729 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,729 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:52,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:52,732 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:52,732 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:52,732 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:52,732 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:52,732 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:52,734 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,734 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,734 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:52,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:52,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:52,735 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,737 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,744 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,744 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,744 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:52,744 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:52,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:52,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:52,744 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:52,744 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,744 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,744 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,744 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,744 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:52,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:52,744 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:52,744 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:52,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:52,845 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:52,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:52,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:52,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:52,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:52,849 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:52,849 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:52,849 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:52,849 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:52,849 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,849 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,851 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,851 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:52,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:52,851 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,851 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:52,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:52,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:52,852 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:52,852 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:52,852 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:52,853 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:52,853 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:52,853 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:52,853 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:52,853 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:52,853 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:52,853 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:52,853 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:52,853 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,853 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:52,853 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:52,855 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,855 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,855 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,855 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:52,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c356b07[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:52,856 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:52,857 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,861 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,861 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:52,861 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:52,861 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:52,861 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,862 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:52,863 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,863 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:52,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,863 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:52,863 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:52,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:52,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:52,865 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:52,865 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:52,865 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:52,865 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:52,866 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:52,867 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,867 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,867 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:52,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:52,868 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,869 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:52,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:52,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:52,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:52,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:52,872 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=549 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/da10ad07a7c3424bb8d5f4633c5ff7be 2023-04-18 18:54:52,872 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,872 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,872 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,872 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,872 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:52,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:52,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:52,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:52,873 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:52,873 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:52,973 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:52,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:52,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:52,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:52,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:52,976 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:52,976 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:52,976 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:52,976 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:52,977 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,977 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,979 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:52,979 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:52,980 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,980 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:52,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:52,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:52,981 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:52,981 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:52,981 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:52,981 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:52,982 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:52,982 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:52,982 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:52,982 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:52,982 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:52,982 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:52,982 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:52,982 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,982 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:52,982 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:52,983 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,983 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,984 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,984 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:52,986 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:52,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36d5229e[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:52,986 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,987 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,987 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:52,987 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,987 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:52,987 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:52,987 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:52,988 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:52,988 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:52,988 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,988 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:52,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:52,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:52,991 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:52,991 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:52,991 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:52,991 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:52,991 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:52,994 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,994 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:52,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,994 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:52,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:52,995 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,996 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,998 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,998 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,998 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:52,998 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:52,998 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:52,998 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:52,998 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:52,998 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:52,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:52,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:52,998 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:52,998 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:52,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:52,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:52,999 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:52,999 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:52,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:52,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:53,099 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:53,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:53,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:53,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:53,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:53,102 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:53,102 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:53,102 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:53,102 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:53,103 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:53,103 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:53,104 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,104 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:53,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:53,104 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,104 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:53,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:53,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:53,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:53,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:53,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:53,106 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:53,106 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:53,106 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:53,106 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:53,106 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:53,106 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:53,106 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:53,106 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:53,106 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:53,106 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:53,106 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:53,106 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:53,106 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:53,106 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,106 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:53,106 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:53,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,108 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:53,108 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:53,108 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:53,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:53,110 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:53,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ddd159a[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:53,110 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:53,112 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:53,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:53,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:53,112 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:53,113 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,113 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:53,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:53,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:53,114 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:53,114 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:53,114 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,114 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:53,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:53,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:53,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:53,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:53,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:53,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:53,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:53,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:53,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:53,117 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:53,117 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:53,117 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:53,117 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:53,117 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:53,119 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:53,119 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:53,119 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:53,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:53,119 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:53,120 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:53,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:53,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,123 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,123 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:53,123 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:53,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:53,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:53,123 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:53,123 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:53,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:53,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:53,123 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:53,124 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,124 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:53,124 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:53,124 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:53,124 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:53,124 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:53,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:53,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:53,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:53,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844153175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:53,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844153198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:53,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844153199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,224 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:53,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:53,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:53,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:53,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:53,227 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:53,227 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:53,227 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:53,227 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:53,228 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:53,228 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:53,230 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,230 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:53,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:53,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:53,231 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,231 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:53,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:53,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:53,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:53,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:53,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:53,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:53,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:53,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:53,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:53,232 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:53,232 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:53,232 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:53,232 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:53,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:53,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:53,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,233 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:53,233 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:53,234 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:53,234 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:53,235 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:53,235 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:53,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:53,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4099924b[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:53,236 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:53,236 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:53,237 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:53,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:53,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:53,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:53,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:53,237 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,237 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:53,247 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,247 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:53,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:53,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:53,247 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:53,247 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:53,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:53,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:53,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:53,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:53,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:53,250 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:53,250 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:53,250 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:53,250 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:53,250 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:53,251 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:53,251 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:53,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:53,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:53,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:53,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:53,251 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:53,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:53,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:53,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:53,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:53,252 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:53,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:53,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:53,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:53,253 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:53,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:53,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:53,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:53,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:53,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:53,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:53,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:53,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:53,256 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:53,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:53,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:53,256 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:53,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:53,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:53,256 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:53,256 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,256 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:53,256 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:53,256 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:53,283 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=549 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/c801ebe22f6849648b750b70c0d3b068 2023-04-18 18:54:53,287 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/936ad3a4f9d24b4fa524b19b3411cb31 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/936ad3a4f9d24b4fa524b19b3411cb31 2023-04-18 18:54:53,291 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/936ad3a4f9d24b4fa524b19b3411cb31, entries=150, sequenceid=549, filesize=11.9 K 2023-04-18 18:54:53,292 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/da10ad07a7c3424bb8d5f4633c5ff7be as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/da10ad07a7c3424bb8d5f4633c5ff7be 2023-04-18 18:54:53,295 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/da10ad07a7c3424bb8d5f4633c5ff7be, entries=150, sequenceid=549, filesize=11.9 K 2023-04-18 18:54:53,296 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/c801ebe22f6849648b750b70c0d3b068 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/c801ebe22f6849648b750b70c0d3b068 2023-04-18 18:54:53,299 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/c801ebe22f6849648b750b70c0d3b068, entries=150, sequenceid=549, filesize=11.9 K 2023-04-18 18:54:53,300 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for e3c3c3367729f1159cdc97339f70376a in 1260ms, sequenceid=549, compaction requested=true 2023-04-18 18:54:53,300 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:53,300 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:53,300 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:54:53,300 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:54:53,300 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:53,300 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:53,301 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:54:53,301 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:54:53,301 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/A is initiating minor compaction (all files) 2023-04-18 18:54:53,301 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/B is initiating minor compaction (all files) 2023-04-18 18:54:53,301 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/A in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:53,301 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/B in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:53,302 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/3a1caf5c45e8436aaf444a09ede78e36, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/0c059abb2b964d91a6540ca35f09667e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/936ad3a4f9d24b4fa524b19b3411cb31] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=36.2 K 2023-04-18 18:54:53,302 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/b8a46de2595a4b0f89e97c3d82d11cb1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/788cfaf5966b432b89987015941589cb, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/da10ad07a7c3424bb8d5f4633c5ff7be] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=36.2 K 2023-04-18 18:54:53,302 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 3a1caf5c45e8436aaf444a09ede78e36, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1681844090558 2023-04-18 18:54:53,302 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b8a46de2595a4b0f89e97c3d82d11cb1, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1681844090558 2023-04-18 18:54:53,302 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 0c059abb2b964d91a6540ca35f09667e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681844090596 2023-04-18 18:54:53,302 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 788cfaf5966b432b89987015941589cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681844090596 2023-04-18 18:54:53,302 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 936ad3a4f9d24b4fa524b19b3411cb31, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=549, earliestPutTs=1681844091729 2023-04-18 18:54:53,303 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting da10ad07a7c3424bb8d5f4633c5ff7be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=549, earliestPutTs=1681844091729 2023-04-18 18:54:53,309 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#B#compaction#249 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:53,309 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#A#compaction#248 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:53,333 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/1bcc18ce547c499fac587737d7482f4c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/1bcc18ce547c499fac587737d7482f4c 2023-04-18 18:54:53,339 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/B of e3c3c3367729f1159cdc97339f70376a into 1bcc18ce547c499fac587737d7482f4c(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:53,339 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:53,339 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/B, priority=13, startTime=1681844093300; duration=0sec 2023-04-18 18:54:53,340 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:53,340 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:54:53,340 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:54:53,341 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/C is initiating minor compaction (all files) 2023-04-18 18:54:53,341 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/C in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:53,341 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bebd2dd5b10c4498a3f9763b643289c2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/2a14f50354f741c1b59bfb04a8e02558, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/c801ebe22f6849648b750b70c0d3b068] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=36.2 K 2023-04-18 18:54:53,341 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting bebd2dd5b10c4498a3f9763b643289c2, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1681844090558 2023-04-18 18:54:53,342 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 2a14f50354f741c1b59bfb04a8e02558, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681844090596 2023-04-18 18:54:53,342 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting c801ebe22f6849648b750b70c0d3b068, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=549, earliestPutTs=1681844091729 2023-04-18 18:54:53,348 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#C#compaction#250 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:53,356 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:53,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:53,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:53,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:53,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:53,359 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:53,359 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:53,359 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:53,359 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:53,360 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:53,360 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:53,361 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,361 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:53,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:53,361 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,361 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:53,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:53,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:53,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:53,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:53,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:53,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:53,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:53,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:53,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:53,363 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:53,363 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:53,363 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-18 18:54:53,363 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:53,363 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:53,363 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:53,364 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:53,364 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:53,364 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:53,743 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/d1360d5d68a94910940a61f4ad6f7b75 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/d1360d5d68a94910940a61f4ad6f7b75 2023-04-18 18:54:53,748 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/A of e3c3c3367729f1159cdc97339f70376a into d1360d5d68a94910940a61f4ad6f7b75(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:53,748 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:53,748 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/A, priority=13, startTime=1681844093300; duration=0sec 2023-04-18 18:54:53,748 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:53,756 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/c8f83b60c70b4a7dbf22ad6768851bcc as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/c8f83b60c70b4a7dbf22ad6768851bcc 2023-04-18 18:54:53,760 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/C of e3c3c3367729f1159cdc97339f70376a into c8f83b60c70b4a7dbf22ad6768851bcc(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:53,760 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:53,760 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/C, priority=13, startTime=1681844093300; duration=0sec 2023-04-18 18:54:53,761 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:53,771 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/03c8473a119d4d1c970867b7270973c3 2023-04-18 18:54:53,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:53,777 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:53,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:53,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844153788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:53,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844153790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:53,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844153891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:53,895 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:53,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844153893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:54,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844154094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:54,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844154096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:54,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844154180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,184 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/743f5565ef7d40549fd364bf130b9d78 2023-04-18 18:54:54,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:54,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844154203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:54,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844154205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:54,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844154396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:54,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844154398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,595 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/041e601490a7465ba23943a14250da5c 2023-04-18 18:54:54,599 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/03c8473a119d4d1c970867b7270973c3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/03c8473a119d4d1c970867b7270973c3 2023-04-18 18:54:54,603 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/03c8473a119d4d1c970867b7270973c3, entries=150, sequenceid=584, filesize=11.9 K 2023-04-18 18:54:54,605 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/743f5565ef7d40549fd364bf130b9d78 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/743f5565ef7d40549fd364bf130b9d78 2023-04-18 18:54:54,609 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/743f5565ef7d40549fd364bf130b9d78, entries=150, sequenceid=584, filesize=11.9 K 2023-04-18 18:54:54,609 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/041e601490a7465ba23943a14250da5c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/041e601490a7465ba23943a14250da5c 2023-04-18 18:54:54,613 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/041e601490a7465ba23943a14250da5c, entries=150, sequenceid=584, filesize=11.9 K 2023-04-18 18:54:54,614 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for e3c3c3367729f1159cdc97339f70376a in 1251ms, sequenceid=584, compaction requested=false 2023-04-18 18:54:54,614 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:54,614 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:54,617 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:54,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:54,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b3b40c[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:54,619 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:54,619 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:54,620 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,621 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:54,622 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,622 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:54,622 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:54,623 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:54,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:54,625 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:54,625 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:54,625 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:54,625 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:54,625 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:54,627 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:54,627 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:54,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:54,627 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:54,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:54,628 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:54,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:54,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:54,629 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:54,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:54,633 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,633 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:54,633 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:54,633 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:54,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:54,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:54,633 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,633 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:54,633 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:54,633 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:54,633 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:54,633 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:54,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:54,633 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:54,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:54,633 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:54,634 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:54,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:54,734 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:54,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:54,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:54,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:54,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:54,737 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:54,737 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:54,737 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:54,737 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:54,737 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:54,737 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:54,739 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:54,739 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:54,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:54,739 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,739 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:54,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:54,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:54,740 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:54,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:54,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:54,740 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:54,740 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:54,740 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:54,741 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:54,741 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:54,741 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:54,741 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:54,741 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-18 18:54:54,741 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:54,741 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:54,741 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:54,741 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:54,741 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:54,741 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:54,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:54,901 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:54,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844094916, max timestamp=1681844094927Num uniques 859; 2023-04-18 18:54:54,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:54:54,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844094916, max timestamp=1681844094927Num uniques 859; 2023-04-18 18:54:54,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:54:54,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844094916, max timestamp=1681844094927Num uniques 859; 2023-04-18 18:54:54,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:54:54,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:54,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844154964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:54,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:54,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844154965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:55,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:55,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844155068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:55,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:55,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844155068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:55,150 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=598 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/608b622b39c44a559520cb0a13a99774 2023-04-18 18:54:55,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:55,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844155271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:55,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:55,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844155272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:55,565 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=598 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/73165cfc62e0405b92d2c64dc21ace3c 2023-04-18 18:54:55,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:55,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844155582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:55,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:55,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844155583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:55,987 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=598 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/b0334c76c3c5439499b61b051d44dae3 2023-04-18 18:54:55,991 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/608b622b39c44a559520cb0a13a99774 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/608b622b39c44a559520cb0a13a99774 2023-04-18 18:54:55,995 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/608b622b39c44a559520cb0a13a99774, entries=150, sequenceid=598, filesize=11.9 K 2023-04-18 18:54:55,996 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/73165cfc62e0405b92d2c64dc21ace3c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/73165cfc62e0405b92d2c64dc21ace3c 2023-04-18 18:54:56,009 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/73165cfc62e0405b92d2c64dc21ace3c, entries=150, sequenceid=598, filesize=11.9 K 2023-04-18 18:54:56,010 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/b0334c76c3c5439499b61b051d44dae3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b0334c76c3c5439499b61b051d44dae3 2023-04-18 18:54:56,016 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b0334c76c3c5439499b61b051d44dae3, entries=150, sequenceid=598, filesize=11.9 K 2023-04-18 18:54:56,017 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for e3c3c3367729f1159cdc97339f70376a in 1276ms, sequenceid=598, compaction requested=true 2023-04-18 18:54:56,017 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:56,017 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:56,017 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:56,017 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:56,017 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,017 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:56,017 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:56,019 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:56,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18edb7b6[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:56,022 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:56,022 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,023 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,023 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:56,023 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,024 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:56,024 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:56,024 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:56,027 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,027 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,027 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:56,027 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:56,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:56,030 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:56,030 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:56,030 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:56,030 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:56,030 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:56,032 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,032 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,032 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:56,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,033 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:56,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:56,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,034 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,038 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:56,038 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:56,038 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:56,038 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:56,038 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,038 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,038 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,038 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,038 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,038 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:56,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:56,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:56,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:56,038 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:56,038 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:56,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:56,091 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-18 18:54:56,092 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:56,092 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:56,092 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:56,092 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:56,092 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:56,092 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:54:56,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:56,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844156105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:56,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844156105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,139 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:56,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:56,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:56,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:56,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:56,142 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:56,142 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:56,142 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:56,142 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:56,142 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,142 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,144 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:56,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:56,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:56,144 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,145 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,145 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:56,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:56,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:56,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:56,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:56,146 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:56,146 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:56,146 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:56,146 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:56,146 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:56,146 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:56,146 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:56,147 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:56,147 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:56,147 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:56,147 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:56,147 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,147 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:56,147 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:56,149 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,149 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,150 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,150 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:56,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:56,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d9900f4[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:56,151 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:56,151 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,154 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,154 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:56,154 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:56,154 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:56,154 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,154 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:56,157 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:56,157 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:56,157 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,157 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:56,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:56,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:56,160 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:56,160 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:56,160 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:56,160 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:56,160 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:56,163 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,163 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,163 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:56,163 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:56,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:56,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,164 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,170 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,170 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,170 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,170 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,170 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:56,170 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:56,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:56,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:56,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:56,170 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:56,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:56,170 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:56,170 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:56,170 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:56,170 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:56,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:56,170 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,194 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:56,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36732 deadline: 1681844156193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,196 DEBUG [Thread-1522] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4137 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:54:56,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:56,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844156210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:56,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844156210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:56,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1681844156210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,212 DEBUG [Thread-1520] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4130 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:54:56,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:56,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844156214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,216 DEBUG [Thread-1524] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4131 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:54:56,270 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:56,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:56,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:56,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:56,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:56,277 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:56,277 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:56,277 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:56,277 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:56,277 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,277 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,279 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,279 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:56,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:56,279 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,279 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:56,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:56,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:56,281 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:56,281 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:56,281 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:56,281 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:56,281 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:56,281 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:56,281 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:56,281 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:56,281 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:56,281 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:56,281 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:56,281 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,282 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:56,282 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:56,283 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,283 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,284 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,284 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:56,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@119ec2c6[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:56,285 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:56,285 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,287 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,287 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:56,287 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:56,287 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:56,287 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,287 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:56,289 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,289 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:56,289 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,289 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:56,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:56,292 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:56,292 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:56,292 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:56,292 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:56,292 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:56,293 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,293 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,294 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:56,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,294 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:56,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:56,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,295 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,297 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:56,298 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:56,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:56,298 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:56,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:56,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:56,298 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,298 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,298 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,298 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,298 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,298 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,298 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,298 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:56,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:56,298 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:56,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:56,399 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:56,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:56,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:56,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:56,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:56,402 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:56,402 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:56,402 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:56,402 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:56,405 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,405 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,407 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,407 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:56,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:56,407 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,407 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:56,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:56,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:56,408 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:56,408 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:56,408 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:56,408 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:56,408 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:56,409 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:56,409 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:56,409 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:56,409 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:56,409 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:56,409 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:56,409 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,409 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:56,409 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:56,411 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,411 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,412 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,412 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:56,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:56,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b97072d[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:56,413 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:56,413 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,414 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:56,415 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,415 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:56,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:56,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:56,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:56,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844156412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:56,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844156413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,416 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:56,416 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,416 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:56,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,416 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:56,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:56,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:56,419 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:56,419 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:56,419 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:56,419 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:56,419 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:56,420 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,420 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,420 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:56,421 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,422 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,425 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,425 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,425 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,425 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,425 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,425 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:56,425 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:56,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:56,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:56,425 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,425 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:56,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:56,425 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:56,426 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:56,426 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:56,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:56,504 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/f85c8bc026d64a7f827731993159122b 2023-04-18 18:54:56,522 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/86c26ddd1281424f9abb50cde3db3d12 2023-04-18 18:54:56,526 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:56,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:56,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:56,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:56,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:56,530 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:56,530 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:56,530 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:56,530 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:56,530 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,530 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,532 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,532 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:56,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:56,532 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,532 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:56,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:56,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:56,534 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:56,534 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:56,534 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:56,534 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:56,534 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:56,534 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:56,534 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:56,534 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:56,534 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:56,534 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:56,534 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:56,534 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,534 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:56,534 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:56,536 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,536 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,537 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,537 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:56,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eb251ee[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:56,538 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:56,538 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,539 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,540 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:56,540 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:56,540 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:56,540 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,540 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:56,543 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:56,543 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:56,543 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,544 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:56,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:56,546 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:56,547 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:56,547 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:56,547 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:56,547 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:56,548 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,548 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:56,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,548 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:56,549 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,550 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,555 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:56,555 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,555 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:56,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:56,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:56,555 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:56,555 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:56,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:56,555 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,555 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:56,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:56,556 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,556 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,556 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,556 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:56,556 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:56,656 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:56,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:56,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:56,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:56,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:56,659 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:56,659 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:56,659 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:56,660 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:56,660 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,660 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,662 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:56,662 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:56,662 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,662 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:56,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:56,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:56,663 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:56,663 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:56,663 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:56,663 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:56,663 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:56,664 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:56,664 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:56,664 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:56,664 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:56,664 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:56,664 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:56,664 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,664 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:56,664 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:56,665 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,665 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,666 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,666 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:56,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bf9606c[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:56,668 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:56,668 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,669 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,669 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:56,669 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:56,669 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:56,669 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,669 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:56,671 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:56,671 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:56,671 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,671 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:56,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:56,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:56,673 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:56,673 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:56,673 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:56,673 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:56,673 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:56,676 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,676 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:56,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,676 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,677 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,678 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,680 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,680 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:56,680 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:56,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:56,680 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:56,680 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,680 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,680 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:56,680 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:56,680 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:56,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:56,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:56,680 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:56,680 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,680 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:56,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:56,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:56,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844156716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:56,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844156717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,780 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:56,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:56,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:56,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:56,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:56,784 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:56,784 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:56,784 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:56,784 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:56,784 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,784 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,785 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:56,785 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:56,786 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,786 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:56,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:56,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:56,787 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:56,787 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:56,787 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:56,787 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:56,787 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:56,787 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:56,787 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:56,787 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:56,788 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:56,788 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:56,788 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:56,788 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,788 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:56,788 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:56,789 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,789 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,789 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,789 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:56,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c960f27[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:56,791 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:56,791 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,793 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,794 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:56,794 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:56,794 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,794 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:56,794 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:56,795 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,795 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:56,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,795 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:56,795 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:56,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:56,797 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:56,798 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:56,798 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:56,798 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:56,798 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:56,799 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,799 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,799 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:56,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,800 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,801 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,803 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,803 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:56,803 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:56,803 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:56,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:56,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:56,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:56,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:56,803 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,803 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:56,804 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:56,804 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,804 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:56,804 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:56,804 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,804 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:56,905 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:56,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:56,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:56,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:56,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:56,908 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:56,908 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:56,908 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:56,908 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:56,908 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,909 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,911 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,911 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:56,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:56,911 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,911 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,912 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:56,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:56,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:56,912 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:56,912 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:56,913 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:56,913 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:56,913 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:56,913 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:56,913 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:56,913 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:56,913 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:56,913 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:56,914 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:56,914 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,914 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:56,914 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:56,915 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,915 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,916 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,916 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:56,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f8bc7f7[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:56,917 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:56,917 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,918 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,918 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:56,918 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:56,918 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:56,919 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,919 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:56,920 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,920 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:56,920 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,920 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:56,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:56,922 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:56,922 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:56,922 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:56,922 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:56,922 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:56,924 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,924 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,925 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:56,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:56,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:56,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:56,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:56,925 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:56,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:56,926 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,928 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:56,928 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:56,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:56,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:56,929 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:56,929 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:56,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:56,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:56,929 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:56,929 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:56,929 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:56,929 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,929 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:56,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:56,929 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:56,929 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:56,929 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:56,934 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/4a3781aa449a4bcaa1fc27c6db83f0db 2023-04-18 18:54:56,939 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/f85c8bc026d64a7f827731993159122b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/f85c8bc026d64a7f827731993159122b 2023-04-18 18:54:56,943 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/f85c8bc026d64a7f827731993159122b, entries=150, sequenceid=634, filesize=11.9 K 2023-04-18 18:54:56,944 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/86c26ddd1281424f9abb50cde3db3d12 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/86c26ddd1281424f9abb50cde3db3d12 2023-04-18 18:54:56,948 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/86c26ddd1281424f9abb50cde3db3d12, entries=150, sequenceid=634, filesize=11.9 K 2023-04-18 18:54:56,949 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/4a3781aa449a4bcaa1fc27c6db83f0db as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/4a3781aa449a4bcaa1fc27c6db83f0db 2023-04-18 18:54:56,958 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/4a3781aa449a4bcaa1fc27c6db83f0db, entries=150, sequenceid=634, filesize=11.9 K 2023-04-18 18:54:56,959 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for e3c3c3367729f1159cdc97339f70376a in 869ms, sequenceid=634, compaction requested=true 2023-04-18 18:54:56,959 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:56,959 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:56,959 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:54:56,959 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:54:56,959 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:56,959 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:56,961 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:54:56,961 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/A is initiating minor compaction (all files) 2023-04-18 18:54:56,961 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:54:56,961 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/A in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:56,961 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/B is initiating minor compaction (all files) 2023-04-18 18:54:56,961 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/d1360d5d68a94910940a61f4ad6f7b75, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/03c8473a119d4d1c970867b7270973c3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/608b622b39c44a559520cb0a13a99774, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/f85c8bc026d64a7f827731993159122b] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=48.2 K 2023-04-18 18:54:56,961 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/B in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:56,962 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/1bcc18ce547c499fac587737d7482f4c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/743f5565ef7d40549fd364bf130b9d78, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/73165cfc62e0405b92d2c64dc21ace3c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/86c26ddd1281424f9abb50cde3db3d12] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=48.2 K 2023-04-18 18:54:56,962 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting d1360d5d68a94910940a61f4ad6f7b75, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=549, earliestPutTs=1681844091729 2023-04-18 18:54:56,962 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 1bcc18ce547c499fac587737d7482f4c, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=549, earliestPutTs=1681844091729 2023-04-18 18:54:56,962 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 03c8473a119d4d1c970867b7270973c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1681844092072 2023-04-18 18:54:56,963 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 743f5565ef7d40549fd364bf130b9d78, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1681844092072 2023-04-18 18:54:56,963 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 608b622b39c44a559520cb0a13a99774, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=598, earliestPutTs=1681844093785 2023-04-18 18:54:56,963 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 73165cfc62e0405b92d2c64dc21ace3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=598, earliestPutTs=1681844093785 2023-04-18 18:54:56,964 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 86c26ddd1281424f9abb50cde3db3d12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1681844094959 2023-04-18 18:54:56,964 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting f85c8bc026d64a7f827731993159122b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1681844094959 2023-04-18 18:54:56,978 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#B#compaction#260 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:56,985 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#A#compaction#261 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:56,995 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/e668d2466fd04cab89ad19ddf779d7d8 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/e668d2466fd04cab89ad19ddf779d7d8 2023-04-18 18:54:57,000 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/B of e3c3c3367729f1159cdc97339f70376a into e668d2466fd04cab89ad19ddf779d7d8(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:57,000 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:57,000 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/B, priority=12, startTime=1681844096959; duration=0sec 2023-04-18 18:54:57,000 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:57,000 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:54:57,002 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:54:57,002 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/C is initiating minor compaction (all files) 2023-04-18 18:54:57,002 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/C in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:57,002 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/c8f83b60c70b4a7dbf22ad6768851bcc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/041e601490a7465ba23943a14250da5c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b0334c76c3c5439499b61b051d44dae3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/4a3781aa449a4bcaa1fc27c6db83f0db] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=48.2 K 2023-04-18 18:54:57,002 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting c8f83b60c70b4a7dbf22ad6768851bcc, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=549, earliestPutTs=1681844091729 2023-04-18 18:54:57,003 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 041e601490a7465ba23943a14250da5c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1681844092072 2023-04-18 18:54:57,003 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b0334c76c3c5439499b61b051d44dae3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=598, earliestPutTs=1681844093785 2023-04-18 18:54:57,003 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 4a3781aa449a4bcaa1fc27c6db83f0db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1681844094959 2023-04-18 18:54:57,012 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#C#compaction#262 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:57,029 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:57,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:57,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:57,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:57,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:57,045 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:57,045 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:57,045 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:57,045 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:57,046 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,046 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:57,049 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:57,049 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:57,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:57,049 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,049 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:57,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:57,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:57,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:57,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:57,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:57,052 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:57,052 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:57,052 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:57,056 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:57,056 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:57,056 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:57,056 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:57,056 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 18:54:57,056 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:57,056 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:57,056 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:57,057 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:57,057 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:57,057 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:57,069 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=645 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/ae9f3e7edc2c4b2d837828a7422892c1 2023-04-18 18:54:57,081 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=645 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/66b5bcbf9538491a967503a5ae61e9f0 2023-04-18 18:54:57,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:57,223 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:57,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844097277, max timestamp=1681844097287Num uniques 859; 2023-04-18 18:54:57,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844097277, max timestamp=1681844097287Num uniques 859; 2023-04-18 18:54:57,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844097277, max timestamp=1681844097287Num uniques 859; 2023-04-18 18:54:57,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:54:57,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:54:57,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:54:57,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:57,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844157316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:57,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844157318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,395 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/2193ed5db95f4ac88c3509f5540ce682 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/2193ed5db95f4ac88c3509f5540ce682 2023-04-18 18:54:57,399 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/A of e3c3c3367729f1159cdc97339f70376a into 2193ed5db95f4ac88c3509f5540ce682(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:57,400 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:57,400 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/A, priority=12, startTime=1681844096959; duration=0sec 2023-04-18 18:54:57,400 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:57,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:57,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844157422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:57,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844157422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,451 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/afc8eb42667c410797456e9e1c56a05d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/afc8eb42667c410797456e9e1c56a05d 2023-04-18 18:54:57,455 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/C of e3c3c3367729f1159cdc97339f70376a into afc8eb42667c410797456e9e1c56a05d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:57,455 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:57,455 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/C, priority=12, startTime=1681844096959; duration=0sec 2023-04-18 18:54:57,455 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:57,492 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=645 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/a9aa45f6aa0947528c1363c2ef98545e 2023-04-18 18:54:57,496 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/ae9f3e7edc2c4b2d837828a7422892c1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/ae9f3e7edc2c4b2d837828a7422892c1 2023-04-18 18:54:57,499 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/ae9f3e7edc2c4b2d837828a7422892c1, entries=150, sequenceid=645, filesize=11.9 K 2023-04-18 18:54:57,500 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/66b5bcbf9538491a967503a5ae61e9f0 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/66b5bcbf9538491a967503a5ae61e9f0 2023-04-18 18:54:57,503 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/66b5bcbf9538491a967503a5ae61e9f0, entries=150, sequenceid=645, filesize=11.9 K 2023-04-18 18:54:57,504 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/a9aa45f6aa0947528c1363c2ef98545e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/a9aa45f6aa0947528c1363c2ef98545e 2023-04-18 18:54:57,507 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/a9aa45f6aa0947528c1363c2ef98545e, entries=150, sequenceid=645, filesize=11.9 K 2023-04-18 18:54:57,508 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e3c3c3367729f1159cdc97339f70376a in 452ms, sequenceid=645, compaction requested=false 2023-04-18 18:54:57,508 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:57,508 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:57,508 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:57,508 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:57,508 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,508 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:57,508 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:57,513 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,513 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:57,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:57,514 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:57,514 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:57,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:57,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:57,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:57,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@135ac4c9[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:57,515 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:57,515 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,516 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,517 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:57,517 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:57,517 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:57,517 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,517 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:57,518 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:57,518 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:57,518 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:57,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:57,518 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:57,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:57,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:57,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:57,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:57,520 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:57,521 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:57,521 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:57,521 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:57,521 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:57,528 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,528 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,528 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:57,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:57,529 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,530 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:57,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,532 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,532 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:57,532 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:57,532 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:57,532 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:57,532 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:57,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:57,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:57,532 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:57,532 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,532 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,532 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,532 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:57,533 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:57,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:57,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:57,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:57,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:57,627 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=147.60 KB heapSize=425.53 KB 2023-04-18 18:54:57,628 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:57,628 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 18:54:57,628 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:57,628 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 18:54:57,628 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:57,628 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 18:54:57,633 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:57,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:57,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:57,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:57,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:57,636 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:57,636 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:57,636 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:57,636 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:57,636 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,637 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:57,637 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:57,637 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:57,638 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,638 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:57,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:57,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:57,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:57,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:57,640 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:57,640 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:57,640 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:57,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:57,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:57,642 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:57,642 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:57,642 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:57,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844157639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,642 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:57,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844157639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,642 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:57,642 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:57,642 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:57,642 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:57,642 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,642 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:57,642 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:57,642 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=686 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/09f6c84a01254a3db7303e40fef481d9 2023-04-18 18:54:57,644 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,644 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,644 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,644 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:57,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:57,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73c03cce[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:57,646 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:57,646 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,648 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,648 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:57,648 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:57,648 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:57,650 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:57,650 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,650 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:57,650 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:57,650 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:57,650 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:57,650 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:57,650 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:57,650 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:57,650 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:57,650 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:57,652 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,652 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:57,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,652 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:57,653 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:57,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:57,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:57,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,655 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:57,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,657 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,657 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:57,657 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:57,657 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,657 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:57,657 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:57,657 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:57,657 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,657 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,657 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:57,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:57,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:57,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:57,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:57,657 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:57,658 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:57,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:57,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:57,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844157743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:57,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844157749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,758 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:57,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:57,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:57,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:57,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:57,761 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:57,761 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:57,761 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:57,761 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:57,762 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,762 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:57,764 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,764 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:57,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:57,765 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,765 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:57,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:57,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:57,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:57,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:57,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:57,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:57,766 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:57,766 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:57,766 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:57,766 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:57,766 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:57,766 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:57,766 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:57,766 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:57,766 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:57,766 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:57,766 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:57,766 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,766 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:57,766 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:57,768 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:57,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:57,768 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:57,769 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,769 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:57,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:57,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:57,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:57,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fd3eaa6[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:57,770 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:57,770 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,771 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,771 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:57,771 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:57,771 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:57,771 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,771 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:57,773 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:57,773 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:57,773 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,773 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:57,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:57,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:57,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:57,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:57,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:57,775 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:57,775 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:57,775 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:57,775 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:57,775 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:57,778 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,778 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:57,778 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:57,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:57,779 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,780 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,782 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,782 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:57,782 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:57,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:57,782 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:57,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:57,782 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:57,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:57,782 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:57,782 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:57,782 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,782 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,782 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:57,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:57,783 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:57,783 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:57,883 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:57,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:57,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:57,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:57,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:57,886 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:57,886 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:57,886 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:57,886 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:57,886 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,886 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:57,887 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:57,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:57,888 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,888 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,888 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:57,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:57,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:57,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:57,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:57,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:57,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:57,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:57,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:57,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:57,890 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:57,890 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:57,890 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:57,890 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:57,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:57,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:57,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:57,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:57,891 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,891 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:57,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:57,892 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,892 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:57,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:57,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:57,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:57,893 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:57,893 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ec630d4[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:57,895 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,896 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,896 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:57,896 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:57,896 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:57,896 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:57,897 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,897 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:57,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:57,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:57,897 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:57,898 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:57,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:57,899 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:57,899 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:57,900 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:57,900 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:57,900 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:57,901 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,901 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:57,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,901 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:57,901 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:57,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,903 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,904 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,904 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:57,904 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:57,904 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:57,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:57,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:57,904 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:57,905 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:57,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:57,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:57,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:57,905 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,905 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:57,905 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:57,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:57,905 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:57,905 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:57,905 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:57,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:57,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844157945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:57,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:57,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844157950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,005 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:58,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:58,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:58,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:58,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:58,009 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:58,009 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:58,009 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:58,009 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:58,010 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:58,010 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:58,012 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,012 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:58,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:58,013 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,013 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:58,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:58,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:58,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:58,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:58,014 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:58,014 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:58,014 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:58,014 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:58,016 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,016 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:58,016 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:58,016 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:58,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:58,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b7fc549[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:58,018 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:58,018 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:58,019 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:58,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:58,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:58,019 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:58,019 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:58,019 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,019 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:58,019 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:58,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:58,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:58,020 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:58,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:58,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:58,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:58,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:58,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:58,022 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:58,022 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:58,023 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:58,023 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:58,023 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:58,025 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:58,025 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:58,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:58,025 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:58,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:58,026 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:58,027 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,029 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,029 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:58,029 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:58,029 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,029 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:58,029 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:58,029 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:58,029 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:58,029 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:58,029 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:58,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:58,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:58,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:58,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:58,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:58,030 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:58,030 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:58,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:58,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:58,054 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=686 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/4e5fe921c8a24927bcaf9a818b59fae1 2023-04-18 18:54:58,065 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=686 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/73d0655bd8f949699d4e6eb0030b3434 2023-04-18 18:54:58,068 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/09f6c84a01254a3db7303e40fef481d9 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/09f6c84a01254a3db7303e40fef481d9 2023-04-18 18:54:58,072 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/09f6c84a01254a3db7303e40fef481d9, entries=150, sequenceid=686, filesize=11.9 K 2023-04-18 18:54:58,073 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/4e5fe921c8a24927bcaf9a818b59fae1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/4e5fe921c8a24927bcaf9a818b59fae1 2023-04-18 18:54:58,078 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/4e5fe921c8a24927bcaf9a818b59fae1, entries=150, sequenceid=686, filesize=11.9 K 2023-04-18 18:54:58,079 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/73d0655bd8f949699d4e6eb0030b3434 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/73d0655bd8f949699d4e6eb0030b3434 2023-04-18 18:54:58,085 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/73d0655bd8f949699d4e6eb0030b3434, entries=150, sequenceid=686, filesize=11.9 K 2023-04-18 18:54:58,086 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~425.48 KB/435696, currentSize=33.54 KB/34350 for e3c3c3367729f1159cdc97339f70376a in 459ms, sequenceid=686, compaction requested=true 2023-04-18 18:54:58,086 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:58,086 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:58,087 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:54:58,087 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:54:58,087 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:58,087 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:58,088 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:54:58,088 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/A is initiating minor compaction (all files) 2023-04-18 18:54:58,088 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/A in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:58,088 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/2193ed5db95f4ac88c3509f5540ce682, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/ae9f3e7edc2c4b2d837828a7422892c1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/09f6c84a01254a3db7303e40fef481d9] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=36.4 K 2023-04-18 18:54:58,088 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:54:58,088 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/B is initiating minor compaction (all files) 2023-04-18 18:54:58,088 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/B in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:58,089 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/e668d2466fd04cab89ad19ddf779d7d8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/66b5bcbf9538491a967503a5ae61e9f0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/4e5fe921c8a24927bcaf9a818b59fae1] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=36.4 K 2023-04-18 18:54:58,090 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 2193ed5db95f4ac88c3509f5540ce682, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1681844094959 2023-04-18 18:54:58,090 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting e668d2466fd04cab89ad19ddf779d7d8, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1681844094959 2023-04-18 18:54:58,090 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting ae9f3e7edc2c4b2d837828a7422892c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=645, earliestPutTs=1681844096095 2023-04-18 18:54:58,090 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 66b5bcbf9538491a967503a5ae61e9f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=645, earliestPutTs=1681844096095 2023-04-18 18:54:58,091 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 4e5fe921c8a24927bcaf9a818b59fae1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1681844097312 2023-04-18 18:54:58,091 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 09f6c84a01254a3db7303e40fef481d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1681844097312 2023-04-18 18:54:58,101 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#B#compaction#269 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:58,105 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#A#compaction#270 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:58,130 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:58,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:58,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:58,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:58,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:58,140 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:58,140 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:58,140 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:58,140 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:58,141 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:58,141 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:58,143 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,143 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:58,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:58,143 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,143 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:58,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:58,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:58,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:58,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:58,152 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:58,152 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:58,152 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:58,152 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:58,152 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:58,152 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:58,152 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:58,153 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-18 18:54:58,153 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:58,153 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:58,153 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:58,153 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:58,153 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:58,153 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:58,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:58,251 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:58,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844098261, max timestamp=1681844098274Num uniques 859; 2023-04-18 18:54:58,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:54:58,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844098261, max timestamp=1681844098274Num uniques 859; 2023-04-18 18:54:58,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844098261, max timestamp=1681844098274Num uniques 859; 2023-04-18 18:54:58,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:54:58,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:54:58,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681844098289, max timestamp=1681844098300Num uniques 1018; 2023-04-18 18:54:58,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-18 18:54:58,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681844098289, max timestamp=1681844098300Num uniques 1018; 2023-04-18 18:54:58,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681844098289, max timestamp=1681844098300Num uniques 1018; 2023-04-18 18:54:58,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-18 18:54:58,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-18 18:54:58,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:58,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844158332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681844098323, max timestamp=1681844098334Num uniques 1077; 2023-04-18 18:54:58,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-18 18:54:58,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681844098323, max timestamp=1681844098334Num uniques 1077; 2023-04-18 18:54:58,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-18 18:54:58,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681844098323, max timestamp=1681844098334Num uniques 1077; 2023-04-18 18:54:58,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-18 18:54:58,393 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681844098379, max timestamp=1681844098391Num uniques 1036; 2023-04-18 18:54:58,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-18 18:54:58,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681844098379, max timestamp=1681844098391Num uniques 1036; 2023-04-18 18:54:58,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-18 18:54:58,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681844098379, max timestamp=1681844098391Num uniques 1036; 2023-04-18 18:54:58,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-18 18:54:58,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681844098432, max timestamp=1681844098441Num uniques 1045; 2023-04-18 18:54:58,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-18 18:54:58,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681844098432, max timestamp=1681844098441Num uniques 1045; 2023-04-18 18:54:58,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681844098432, max timestamp=1681844098441Num uniques 1045; 2023-04-18 18:54:58,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-18 18:54:58,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-18 18:54:58,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:58,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844158470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:58,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844158471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,539 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/fe6e3c82871a4f9699ed2db672aa33ec as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/fe6e3c82871a4f9699ed2db672aa33ec 2023-04-18 18:54:58,540 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/eca997c5eb374839bdd3c28c6f99349c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/eca997c5eb374839bdd3c28c6f99349c 2023-04-18 18:54:58,544 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/A of e3c3c3367729f1159cdc97339f70376a into eca997c5eb374839bdd3c28c6f99349c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:58,544 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:58,544 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/A, priority=13, startTime=1681844098086; duration=0sec 2023-04-18 18:54:58,544 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:58,544 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:54:58,545 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/B of e3c3c3367729f1159cdc97339f70376a into fe6e3c82871a4f9699ed2db672aa33ec(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:58,545 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:58,545 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/B, priority=13, startTime=1681844098087; duration=0sec 2023-04-18 18:54:58,545 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:58,545 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:54:58,545 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/C is initiating minor compaction (all files) 2023-04-18 18:54:58,546 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/C in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:58,546 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/afc8eb42667c410797456e9e1c56a05d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/a9aa45f6aa0947528c1363c2ef98545e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/73d0655bd8f949699d4e6eb0030b3434] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=36.4 K 2023-04-18 18:54:58,546 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting afc8eb42667c410797456e9e1c56a05d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1681844094959 2023-04-18 18:54:58,546 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting a9aa45f6aa0947528c1363c2ef98545e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=645, earliestPutTs=1681844096095 2023-04-18 18:54:58,547 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 73d0655bd8f949699d4e6eb0030b3434, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1681844097312 2023-04-18 18:54:58,552 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#C#compaction#272 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:58,566 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/e73c2c866c6040bbbb6d1204224ef1d6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/e73c2c866c6040bbbb6d1204224ef1d6 2023-04-18 18:54:58,572 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/C of e3c3c3367729f1159cdc97339f70376a into e73c2c866c6040bbbb6d1204224ef1d6(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:58,572 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:58,572 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/C, priority=13, startTime=1681844098087; duration=0sec 2023-04-18 18:54:58,572 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:58,581 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:58,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:58,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844158574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844158574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,582 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/40259220b7a641238ee19f2f74b78d9b 2023-04-18 18:54:58,596 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/35bc2754f92f450abe915a8fb047c0e6 2023-04-18 18:54:58,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:58,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844158782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:58,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:58,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844158782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,005 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/a1fba916d7cd4e1eaca275147f7133dc 2023-04-18 18:54:59,010 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/40259220b7a641238ee19f2f74b78d9b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/40259220b7a641238ee19f2f74b78d9b 2023-04-18 18:54:59,014 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/40259220b7a641238ee19f2f74b78d9b, entries=100, sequenceid=694, filesize=9.5 K 2023-04-18 18:54:59,014 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/35bc2754f92f450abe915a8fb047c0e6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/35bc2754f92f450abe915a8fb047c0e6 2023-04-18 18:54:59,020 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/35bc2754f92f450abe915a8fb047c0e6, entries=100, sequenceid=694, filesize=9.5 K 2023-04-18 18:54:59,021 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/a1fba916d7cd4e1eaca275147f7133dc as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/a1fba916d7cd4e1eaca275147f7133dc 2023-04-18 18:54:59,024 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/a1fba916d7cd4e1eaca275147f7133dc, entries=100, sequenceid=694, filesize=9.5 K 2023-04-18 18:54:59,025 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for e3c3c3367729f1159cdc97339f70376a in 872ms, sequenceid=694, compaction requested=false 2023-04-18 18:54:59,025 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:59,025 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:59,025 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:59,025 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:59,025 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,025 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:59,025 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:59,028 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:59,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:59,028 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:59,028 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,028 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:59,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:59,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:59,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:59,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@192bb2a4[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:59,030 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:59,030 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,032 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,032 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:59,032 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:59,032 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,032 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:59,032 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:59,033 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:59,033 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:59,033 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,034 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:59,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:59,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:59,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:59,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:59,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:59,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:59,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:59,036 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:59,036 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:59,036 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:59,036 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:59,036 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:59,038 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,038 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,038 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:59,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:59,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:59,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:59,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:59,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:59,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,039 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:59,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,040 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:59,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,042 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,042 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:59,042 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:59,042 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:59,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:59,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:59,042 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,043 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,043 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,043 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,043 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:59,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:59,042 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:59,043 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:59,043 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:59,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:59,086 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=154.31 KB heapSize=437.39 KB 2023-04-18 18:54:59,086 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:59,086 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 18:54:59,087 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:59,087 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 18:54:59,087 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:59,087 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 18:54:59,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:59,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844159091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:59,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844159092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,143 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:59,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:59,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:59,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:59,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:59,146 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:59,146 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:59,146 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:59,146 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:59,146 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,146 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,149 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,149 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:59,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:59,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:59,149 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,149 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:59,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:59,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:59,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:59,150 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:59,150 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:59,150 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:59,150 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:59,152 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,152 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:59,152 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,152 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:59,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:59,154 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:59,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9bdfdfd[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:59,154 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,155 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,155 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:59,155 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,155 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:59,155 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:59,155 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:59,159 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,159 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:59,160 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:59,160 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:59,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:59,162 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:59,162 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:59,162 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:59,162 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:59,162 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:59,163 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,163 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:59,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,164 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:59,164 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,165 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,168 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,168 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:59,168 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:59,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:59,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:59,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:59,168 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:59,168 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,168 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:59,168 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,168 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:59,168 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:59,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:59,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:59,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:59,168 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,168 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,168 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:59,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844159194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,198 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:59,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844159196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,268 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:59,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:59,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:59,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:59,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:59,272 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:59,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:59,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:59,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:59,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,274 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,274 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:59,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:59,274 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,274 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:59,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:59,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:59,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:59,275 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:59,276 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:59,276 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:59,276 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:59,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:59,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:59,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:59,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:59,278 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,278 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:59,279 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,279 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:59,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:59,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16e23c8e[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:59,280 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:59,280 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,281 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:59,282 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,282 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:59,283 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,283 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:59,283 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:59,283 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:59,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:59,286 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:59,286 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:59,286 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:59,286 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:59,286 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:59,287 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,287 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:59,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,288 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:59,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:59,288 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,289 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:59,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,293 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:59,293 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:59,293 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,293 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:59,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:59,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:59,293 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,293 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,293 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,293 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,293 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,293 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:59,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:59,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:59,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:59,294 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:59,294 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:59,394 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:59,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:59,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:59,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:59,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:59,398 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:59,398 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:59,398 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:59,398 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:59,398 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,398 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,399 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,399 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:59,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:59,400 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,400 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:59,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:59,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:59,401 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:59,401 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:59,401 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:59,401 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:59,401 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:59,401 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:59,401 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:59,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:59,401 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:59,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844159399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,401 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:59,401 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:59,401 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:59,402 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,402 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:59,402 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:59,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:54:59,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844159400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,403 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,403 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:59,403 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,403 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:59,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:59,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a2f696[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:59,405 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:59,405 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,406 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,406 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:59,406 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:59,406 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,406 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:59,406 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:59,409 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:59,411 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:59,412 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:59,412 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:59,412 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:59,412 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:59,413 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,413 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:59,413 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:59,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:59,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:59,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:59,414 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,415 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,417 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,417 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,417 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,417 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:59,417 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:59,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:59,417 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:59,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:59,417 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:59,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:59,417 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:59,417 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,417 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,417 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:59,418 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:59,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:59,495 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=804 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/ebb99bc4c27d474a9766adea3723547b 2023-04-18 18:54:59,505 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=804 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/849b0310f4a24753a4102462ae7775b1 2023-04-18 18:54:59,516 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=804 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/be08611e9fb64373a733fdd0e40b1f51 2023-04-18 18:54:59,518 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:59,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:59,520 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/ebb99bc4c27d474a9766adea3723547b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/ebb99bc4c27d474a9766adea3723547b 2023-04-18 18:54:59,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:59,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:59,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:59,523 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:59,523 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:59,523 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:59,523 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:59,524 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,524 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,524 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/ebb99bc4c27d474a9766adea3723547b, entries=150, sequenceid=804, filesize=11.9 K 2023-04-18 18:54:59,525 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/849b0310f4a24753a4102462ae7775b1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/849b0310f4a24753a4102462ae7775b1 2023-04-18 18:54:59,525 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,525 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:59,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:59,526 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,526 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:59,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:59,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:59,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:59,527 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:59,527 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:59,527 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:59,528 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:59,528 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:59,528 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:59,528 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:59,528 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:59,528 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:59,528 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:54:59,528 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:54:59,528 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,528 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:54:59,528 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:54:59,529 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/849b0310f4a24753a4102462ae7775b1, entries=150, sequenceid=804, filesize=11.9 K 2023-04-18 18:54:59,530 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,530 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:59,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:59,530 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,530 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:54:59,531 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/be08611e9fb64373a733fdd0e40b1f51 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/be08611e9fb64373a733fdd0e40b1f51 2023-04-18 18:54:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:59,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:54:59,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47d8cad4[Count = 0] remaining members to acquire global barrier 2023-04-18 18:54:59,532 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:54:59,532 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,533 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:54:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:54:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:54:59,533 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,533 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:54:59,535 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/be08611e9fb64373a733fdd0e40b1f51, entries=150, sequenceid=804, filesize=11.9 K 2023-04-18 18:54:59,535 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,535 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:54:59,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,535 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~437.34 KB/447840, currentSize=26.84 KB/27480 for e3c3c3367729f1159cdc97339f70376a in 449ms, sequenceid=804, compaction requested=true 2023-04-18 18:54:59,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:59,535 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:59,536 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:59,536 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:54:59,536 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:59,536 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:59,536 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:54:59,536 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:54:59,536 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:59,537 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34966 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:54:59,537 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/A is initiating minor compaction (all files) 2023-04-18 18:54:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,537 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/A in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:59,537 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34966 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:54:59,537 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/eca997c5eb374839bdd3c28c6f99349c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/40259220b7a641238ee19f2f74b78d9b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/ebb99bc4c27d474a9766adea3723547b] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=34.1 K 2023-04-18 18:54:59,537 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/B is initiating minor compaction (all files) 2023-04-18 18:54:59,537 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/B in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:59,537 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/fe6e3c82871a4f9699ed2db672aa33ec, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/35bc2754f92f450abe915a8fb047c0e6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/849b0310f4a24753a4102462ae7775b1] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=34.1 K 2023-04-18 18:54:59,538 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting fe6e3c82871a4f9699ed2db672aa33ec, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1681844097312 2023-04-18 18:54:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,538 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 35bc2754f92f450abe915a8fb047c0e6, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1681844097631 2023-04-18 18:54:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:59,538 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 849b0310f4a24753a4102462ae7775b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=804, earliestPutTs=1681844098461 2023-04-18 18:54:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:54:59,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:54:59,540 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:54:59,540 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:54:59,540 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:54:59,540 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:54:59,540 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting eca997c5eb374839bdd3c28c6f99349c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1681844097312 2023-04-18 18:54:59,540 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:54:59,541 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 40259220b7a641238ee19f2f74b78d9b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1681844097631 2023-04-18 18:54:59,541 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting ebb99bc4c27d474a9766adea3723547b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=804, earliestPutTs=1681844098461 2023-04-18 18:54:59,546 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,546 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:59,546 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:54:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:54:59,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:54:59,547 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,549 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:54:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:54:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:54:59,556 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,556 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,556 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,556 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,556 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,556 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:54:59,556 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,556 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:54:59,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:54:59,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:54:59,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:54:59,556 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:59,556 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:54:59,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:59,556 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:54:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:59,557 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:54:59,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:54:59,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:54:59,561 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#A#compaction#278 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:59,577 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#B#compaction#279 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:59,589 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/0a4cd85f63a343cfb1f6d703a302a032 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/0a4cd85f63a343cfb1f6d703a302a032 2023-04-18 18:54:59,594 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/B of e3c3c3367729f1159cdc97339f70376a into 0a4cd85f63a343cfb1f6d703a302a032(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:59,594 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:59,594 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/B, priority=13, startTime=1681844099536; duration=0sec 2023-04-18 18:54:59,595 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:54:59,595 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:54:59,595 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34966 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:54:59,596 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/C is initiating minor compaction (all files) 2023-04-18 18:54:59,596 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/C in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:59,596 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/e73c2c866c6040bbbb6d1204224ef1d6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/a1fba916d7cd4e1eaca275147f7133dc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/be08611e9fb64373a733fdd0e40b1f51] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=34.1 K 2023-04-18 18:54:59,596 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting e73c2c866c6040bbbb6d1204224ef1d6, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1681844097312 2023-04-18 18:54:59,596 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting a1fba916d7cd4e1eaca275147f7133dc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1681844097631 2023-04-18 18:54:59,597 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting be08611e9fb64373a733fdd0e40b1f51, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=804, earliestPutTs=1681844098461 2023-04-18 18:54:59,602 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#C#compaction#280 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:54:59,612 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/5f5851c62aa64aec8bd1ce9400719acc as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/5f5851c62aa64aec8bd1ce9400719acc 2023-04-18 18:54:59,617 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/C of e3c3c3367729f1159cdc97339f70376a into 5f5851c62aa64aec8bd1ce9400719acc(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:59,617 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:59,617 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/C, priority=13, startTime=1681844099536; duration=0sec 2023-04-18 18:54:59,617 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:59,657 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:54:59,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:54:59,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:54:59,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:54:59,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:54:59,661 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:54:59,661 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:59,661 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:54:59,661 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:54:59,661 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,661 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,662 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:54:59,662 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:54:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:54:59,663 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:54:59,663 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:54:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:54:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:54:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:54:59,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:54:59,664 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:54:59,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:54:59,664 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:54:59,664 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:54:59,664 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:54:59,665 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:54:59,665 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:54:59,665 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:54:59,665 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:54:59,665 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-18 18:54:59,665 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:54:59,665 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:59,665 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:54:59,665 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:59,665 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:54:59,665 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:54:59,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:54:59,706 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:54:59,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844099715, max timestamp=1681844099722Num uniques 859; 2023-04-18 18:54:59,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:54:59,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844099715, max timestamp=1681844099722Num uniques 859; 2023-04-18 18:54:59,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:54:59,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844099715, max timestamp=1681844099722Num uniques 859; 2023-04-18 18:54:59,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:54:59,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681844099759, max timestamp=1681844099764Num uniques 1018; 2023-04-18 18:54:59,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-18 18:54:59,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681844099759, max timestamp=1681844099764Num uniques 1018; 2023-04-18 18:54:59,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-18 18:54:59,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681844099759, max timestamp=1681844099764Num uniques 1018; 2023-04-18 18:54:59,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-18 18:54:59,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681844099811, max timestamp=1681844099816Num uniques 1027; 2023-04-18 18:54:59,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681844099811, max timestamp=1681844099816Num uniques 1027; 2023-04-18 18:54:59,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681844099811, max timestamp=1681844099816Num uniques 1027; 2023-04-18 18:54:59,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 18:54:59,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 18:54:59,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 18:54:59,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8506, totalHeapSize=19584, min timestamp=1681844099836, max timestamp=1681844099842Num uniques 1036; 2023-04-18 18:54:59,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8506, totalHeapSize=19584, min timestamp=1681844099836, max timestamp=1681844099842Num uniques 1036; 2023-04-18 18:54:59,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8508, totalHeapSize=19584, min timestamp=1681844099836, max timestamp=1681844099842Num uniques 1036; 2023-04-18 18:54:59,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47436, new segment data size=8506, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-18 18:54:59,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47436, new segment data size=8506, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-18 18:54:59,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47438, new segment data size=8508, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-18 18:54:59,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681844099860, max timestamp=1681844099867Num uniques 1045; 2023-04-18 18:54:59,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-18 18:54:59,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681844099860, max timestamp=1681844099867Num uniques 1045; 2023-04-18 18:54:59,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-18 18:54:59,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681844099860, max timestamp=1681844099867Num uniques 1045; 2023-04-18 18:54:59,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-18 18:54:59,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681844099886, max timestamp=1681844099897Num uniques 1054; 2023-04-18 18:54:59,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-18 18:54:59,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681844099886, max timestamp=1681844099897Num uniques 1054; 2023-04-18 18:54:59,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-18 18:54:59,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681844099886, max timestamp=1681844099897Num uniques 1054; 2023-04-18 18:54:59,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-18 18:54:59,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681844099917, max timestamp=1681844099941Num uniques 1063; 2023-04-18 18:54:59,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-18 18:54:59,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681844099917, max timestamp=1681844099941Num uniques 1063; 2023-04-18 18:54:59,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-18 18:54:59,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681844099917, max timestamp=1681844099941Num uniques 1063; 2023-04-18 18:54:59,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-18 18:54:59,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681844099959, max timestamp=1681844099964Num uniques 1022; 2023-04-18 18:54:59,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-18 18:54:59,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681844099959, max timestamp=1681844099964Num uniques 1022; 2023-04-18 18:54:59,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681844099959, max timestamp=1681844099964Num uniques 1022; 2023-04-18 18:54:59,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-18 18:54:59,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-18 18:54:59,985 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/98ef8efecb4d4a7596d9b1de23a4c209 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/98ef8efecb4d4a7596d9b1de23a4c209 2023-04-18 18:54:59,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681844099971, max timestamp=1681844099987Num uniques 1031; 2023-04-18 18:54:59,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681844099971, max timestamp=1681844099987Num uniques 1031; 2023-04-18 18:54:59,989 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/A of e3c3c3367729f1159cdc97339f70376a into 98ef8efecb4d4a7596d9b1de23a4c209(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:54:59,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-18 18:54:59,990 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:54:59,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-18 18:54:59,990 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/A, priority=13, startTime=1681844099536; duration=0sec 2023-04-18 18:54:59,990 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:54:59,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681844099971, max timestamp=1681844099987Num uniques 1031; 2023-04-18 18:54:59,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-18 18:55:00,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681844100004, max timestamp=1681844100013Num uniques 1040; 2023-04-18 18:55:00,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-18 18:55:00,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681844100004, max timestamp=1681844100013Num uniques 1040; 2023-04-18 18:55:00,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681844100004, max timestamp=1681844100013Num uniques 1040; 2023-04-18 18:55:00,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-18 18:55:00,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-18 18:55:00,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681844100035, max timestamp=1681844100040Num uniques 1099; 2023-04-18 18:55:00,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-18 18:55:00,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681844100035, max timestamp=1681844100040Num uniques 1099; 2023-04-18 18:55:00,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-18 18:55:00,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681844100035, max timestamp=1681844100040Num uniques 1099; 2023-04-18 18:55:00,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-18 18:55:00,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681844100060, max timestamp=1681844100066Num uniques 1108; 2023-04-18 18:55:00,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-18 18:55:00,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681844100060, max timestamp=1681844100066Num uniques 1108; 2023-04-18 18:55:00,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-18 18:55:00,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681844100060, max timestamp=1681844100066Num uniques 1108; 2023-04-18 18:55:00,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-18 18:55:00,073 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=813 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/02952e7825904e9a870504f4eba9bf03 2023-04-18 18:55:00,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681844100085, max timestamp=1681844100094Num uniques 1017; 2023-04-18 18:55:00,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-18 18:55:00,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681844100085, max timestamp=1681844100094Num uniques 1017; 2023-04-18 18:55:00,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681844100085, max timestamp=1681844100094Num uniques 1017; 2023-04-18 18:55:00,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-18 18:55:00,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-18 18:55:00,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681844100113, max timestamp=1681844100120Num uniques 1076; 2023-04-18 18:55:00,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-18 18:55:00,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681844100113, max timestamp=1681844100120Num uniques 1076; 2023-04-18 18:55:00,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681844100113, max timestamp=1681844100120Num uniques 1076; 2023-04-18 18:55:00,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-18 18:55:00,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-18 18:55:00,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681844100138, max timestamp=1681844100142Num uniques 1035; 2023-04-18 18:55:00,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-18 18:55:00,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681844100138, max timestamp=1681844100142Num uniques 1035; 2023-04-18 18:55:00,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-18 18:55:00,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681844100138, max timestamp=1681844100142Num uniques 1035; 2023-04-18 18:55:00,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-18 18:55:00,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681844100154, max timestamp=1681844100166Num uniques 1044; 2023-04-18 18:55:00,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-18 18:55:00,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681844100154, max timestamp=1681844100166Num uniques 1044; 2023-04-18 18:55:00,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-18 18:55:00,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681844100154, max timestamp=1681844100166Num uniques 1044; 2023-04-18 18:55:00,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-18 18:55:00,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681844100186, max timestamp=1681844100194Num uniques 1053; 2023-04-18 18:55:00,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-18 18:55:00,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681844100186, max timestamp=1681844100195Num uniques 1053; 2023-04-18 18:55:00,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681844100186, max timestamp=1681844100195Num uniques 1053; 2023-04-18 18:55:00,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-18 18:55:00,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-18 18:55:00,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1681844100209, max timestamp=1681844100218Num uniques 1012; 2023-04-18 18:55:00,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1681844100209, max timestamp=1681844100218Num uniques 1012; 2023-04-18 18:55:00,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1681844100213, max timestamp=1681844100218Num uniques 1012; 2023-04-18 18:55:00,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-04-18 18:55:00,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=7412, suffix heap size=133216,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=162 2023-04-18 18:55:00,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-04-18 18:55:00,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1681844100230, max timestamp=1681844100235Num uniques 1021; 2023-04-18 18:55:00,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=12406, suffix heap size=134152,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=271 2023-04-18 18:55:00,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1681844100230, max timestamp=1681844100235Num uniques 1071; 2023-04-18 18:55:00,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1681844100230, max timestamp=1681844100235Num uniques 1071; 2023-04-18 18:55:00,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=12406, suffix heap size=139352,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=271 2023-04-18 18:55:00,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=12406, suffix heap size=139352,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=271 2023-04-18 18:55:00,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:00,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 526 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844160250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1681844100244, max timestamp=1681844100251Num uniques 1130; 2023-04-18 18:55:00,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=12820, suffix heap size=145488,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=280 2023-04-18 18:55:00,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1681844100244, max timestamp=1681844100251Num uniques 1130; 2023-04-18 18:55:00,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=12820, suffix heap size=145488,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=280 2023-04-18 18:55:00,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1681844100244, max timestamp=1681844100251Num uniques 1130; 2023-04-18 18:55:00,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=12820, suffix heap size=145488,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=280 2023-04-18 18:55:00,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:00,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36776 deadline: 1681844160265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:00,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 452 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844160266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1681844100262, max timestamp=1681844100267Num uniques 1139; 2023-04-18 18:55:00,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=8654, suffix heap size=146424,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=189 2023-04-18 18:55:00,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1681844100262, max timestamp=1681844100267Num uniques 1139; 2023-04-18 18:55:00,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=8654, suffix heap size=146424,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=189 2023-04-18 18:55:00,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1681844100262, max timestamp=1681844100267Num uniques 1139; 2023-04-18 18:55:00,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=8654, suffix heap size=146424,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=189 2023-04-18 18:55:00,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1681844100284, max timestamp=1681844100290Num uniques 1048; 2023-04-18 18:55:00,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=11358, suffix heap size=136960,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=248 2023-04-18 18:55:00,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1681844100284, max timestamp=1681844100290Num uniques 1048; 2023-04-18 18:55:00,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=11358, suffix heap size=136960,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=248 2023-04-18 18:55:00,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1681844100284, max timestamp=1681844100290Num uniques 1048; 2023-04-18 18:55:00,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=11358, suffix heap size=136960,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=248 2023-04-18 18:55:00,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1681844100311, max timestamp=1681844100322Num uniques 1107; 2023-04-18 18:55:00,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1681844100311, max timestamp=1681844100322Num uniques 1107; 2023-04-18 18:55:00,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1681844100311, max timestamp=1681844100322Num uniques 1107; 2023-04-18 18:55:00,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-18 18:55:00,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-18 18:55:00,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-18 18:55:00,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1681844100346, max timestamp=1681844100356Num uniques 1066; 2023-04-18 18:55:00,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=12176, suffix heap size=138832,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=266 2023-04-18 18:55:00,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1681844100346, max timestamp=1681844100356Num uniques 1066; 2023-04-18 18:55:00,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1681844100346, max timestamp=1681844100356Num uniques 1066; 2023-04-18 18:55:00,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=12176, suffix heap size=138832,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=266 2023-04-18 18:55:00,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=12176, suffix heap size=138832,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=266 2023-04-18 18:55:00,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:00,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 533 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844160394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:00,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 456 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844160394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:00,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1681844100389, max timestamp=1681844100396Num uniques 1125; 2023-04-18 18:55:00,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844160395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=10300, suffix heap size=144968,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=225 2023-04-18 18:55:00,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1681844100389, max timestamp=1681844100396Num uniques 1125; 2023-04-18 18:55:00,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=10300, suffix heap size=144968,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=225 2023-04-18 18:55:00,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1681844100389, max timestamp=1681844100396Num uniques 1125; 2023-04-18 18:55:00,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=10300, suffix heap size=144968,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=225 2023-04-18 18:55:00,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8424, totalHeapSize=19376, min timestamp=1681844100410, max timestamp=1681844100422Num uniques 1084; 2023-04-18 18:55:00,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=8424, suffix heap size=140704,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=184 2023-04-18 18:55:00,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8424, totalHeapSize=19376, min timestamp=1681844100410, max timestamp=1681844100422Num uniques 1084; 2023-04-18 18:55:00,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=8424, suffix heap size=140704,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=184 2023-04-18 18:55:00,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8424, totalHeapSize=19376, min timestamp=1681844100410, max timestamp=1681844100422Num uniques 1084; 2023-04-18 18:55:00,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=8424, suffix heap size=140704,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=184 2023-04-18 18:55:00,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=193, cellSize=8838, totalHeapSize=20312, min timestamp=1681844100441, max timestamp=1681844100450Num uniques 1043; 2023-04-18 18:55:00,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47768, new segment data size=8838, suffix heap size=136440,new segment heap size=20312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=193 2023-04-18 18:55:00,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=193, cellSize=8838, totalHeapSize=20312, min timestamp=1681844100441, max timestamp=1681844100450Num uniques 1043; 2023-04-18 18:55:00,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=193, cellSize=8838, totalHeapSize=20312, min timestamp=1681844100441, max timestamp=1681844100450Num uniques 1043; 2023-04-18 18:55:00,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47768, new segment data size=8838, suffix heap size=136440,new segment heap size=20312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=193 2023-04-18 18:55:00,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47768, new segment data size=8838, suffix heap size=136440,new segment heap size=20312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=193 2023-04-18 18:55:00,484 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=813 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/4f147476b46043eda5194cae866d3f48 2023-04-18 18:55:00,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1681844100483, max timestamp=1681844100496Num uniques 1052; 2023-04-18 18:55:00,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1681844100483, max timestamp=1681844100496Num uniques 1052; 2023-04-18 18:55:00,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48180, new segment data size=6960, suffix heap size=137376,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1052, new segment cells count=152 2023-04-18 18:55:00,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1681844100483, max timestamp=1681844100496Num uniques 1052; 2023-04-18 18:55:00,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48180, new segment data size=6960, suffix heap size=137376,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1052, new segment cells count=152 2023-04-18 18:55:00,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48180, new segment data size=6960, suffix heap size=137376,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1052, new segment cells count=152 2023-04-18 18:55:00,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1681844100507, max timestamp=1681844100511Num uniques 1011; 2023-04-18 18:55:00,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681844100507, max timestamp=1681844100511Num uniques 1011; 2023-04-18 18:55:00,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46296, new segment data size=9656, suffix heap size=133112,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=211 2023-04-18 18:55:00,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1681844100507, max timestamp=1681844100511Num uniques 1011; 2023-04-18 18:55:00,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46296, new segment data size=6870, suffix heap size=133112,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=150 2023-04-18 18:55:00,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46296, new segment data size=9656, suffix heap size=133112,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=211 2023-04-18 18:55:00,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=320, cellSize=14650, totalHeapSize=33520, min timestamp=1681844100519, max timestamp=1681844100525Num uniques 1070; 2023-04-18 18:55:00,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49000, new segment data size=14650, suffix heap size=139248,new segment heap size=33520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=320 2023-04-18 18:55:00,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=320, cellSize=14650, totalHeapSize=33520, min timestamp=1681844100519, max timestamp=1681844100525Num uniques 1070; 2023-04-18 18:55:00,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49000, new segment data size=14650, suffix heap size=139248,new segment heap size=33520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=320 2023-04-18 18:55:00,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=320, cellSize=14650, totalHeapSize=33520, min timestamp=1681844100519, max timestamp=1681844100525Num uniques 1009; 2023-04-18 18:55:00,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=14650, suffix heap size=132904,new segment heap size=33520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=320 2023-04-18 18:55:00,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:00,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36776 deadline: 1681844160540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:00,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 545 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844160540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1681844100537, max timestamp=1681844100543Num uniques 1179; 2023-04-18 18:55:00,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53994, new segment data size=8194, suffix heap size=150584,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1179, new segment cells count=179 2023-04-18 18:55:00,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681844100537, max timestamp=1681844100543Num uniques 1179; 2023-04-18 18:55:00,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53994, new segment data size=6870, suffix heap size=150584,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1179, new segment cells count=150 2023-04-18 18:55:00,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681844100537, max timestamp=1681844100543Num uniques 1179; 2023-04-18 18:55:00,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53994, new segment data size=6870, suffix heap size=150584,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1179, new segment cells count=150 2023-04-18 18:55:00,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1681844100561, max timestamp=1681844100566Num uniques 1038; 2023-04-18 18:55:00,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=10898, suffix heap size=135920,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=238 2023-04-18 18:55:00,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1681844100561, max timestamp=1681844100566Num uniques 1009; 2023-04-18 18:55:00,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10898, suffix heap size=132904,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=238 2023-04-18 18:55:00,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1681844100561, max timestamp=1681844100566Num uniques 1009; 2023-04-18 18:55:00,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10898, suffix heap size=132904,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=238 2023-04-18 18:55:00,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=247, cellSize=11312, totalHeapSize=25928, min timestamp=1681844100580, max timestamp=1681844100587Num uniques 1097; 2023-04-18 18:55:00,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=11312, suffix heap size=142056,new segment heap size=25928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=247 2023-04-18 18:55:00,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=247, cellSize=11312, totalHeapSize=25928, min timestamp=1681844100580, max timestamp=1681844100587Num uniques 1097; 2023-04-18 18:55:00,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=11312, suffix heap size=142056,new segment heap size=25928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=247 2023-04-18 18:55:00,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=247, cellSize=11312, totalHeapSize=25928, min timestamp=1681844100580, max timestamp=1681844100587Num uniques 1097; 2023-04-18 18:55:00,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=11312, suffix heap size=142056,new segment heap size=25928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=247 2023-04-18 18:55:00,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:00,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 485 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844160610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:00,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36732 deadline: 1681844160612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=206, cellSize=9430, totalHeapSize=21664, min timestamp=1681844100600, max timestamp=1681844100612Num uniques 1106; 2023-04-18 18:55:00,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=206, cellSize=9430, totalHeapSize=21664, min timestamp=1681844100600, max timestamp=1681844100612Num uniques 1106; 2023-04-18 18:55:00,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50650, new segment data size=9430, suffix heap size=142992,new segment heap size=21664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1106, new segment cells count=206 2023-04-18 18:55:00,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=206, cellSize=9430, totalHeapSize=21664, min timestamp=1681844100600, max timestamp=1681844100612Num uniques 1106; 2023-04-18 18:55:00,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50650, new segment data size=9430, suffix heap size=142992,new segment heap size=21664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1106, new segment cells count=206 2023-04-18 18:55:00,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50650, new segment data size=9430, suffix heap size=142992,new segment heap size=21664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1106, new segment cells count=206 2023-04-18 18:55:00,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7550, totalHeapSize=17400, min timestamp=1681844100632, max timestamp=1681844100657Num uniques 1065; 2023-04-18 18:55:00,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48770, new segment data size=7550, suffix heap size=138728,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1065, new segment cells count=165 2023-04-18 18:55:00,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7550, totalHeapSize=17400, min timestamp=1681844100632, max timestamp=1681844100657Num uniques 1065; 2023-04-18 18:55:00,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7550, totalHeapSize=17400, min timestamp=1681844100632, max timestamp=1681844100657Num uniques 1065; 2023-04-18 18:55:00,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48770, new segment data size=7550, suffix heap size=138728,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1065, new segment cells count=165 2023-04-18 18:55:00,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48770, new segment data size=7550, suffix heap size=138728,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1065, new segment cells count=165 2023-04-18 18:55:00,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1681844100672, max timestamp=1681844100677Num uniques 1024; 2023-04-18 18:55:00,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46894, new segment data size=10254, suffix heap size=134464,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1024, new segment cells count=224 2023-04-18 18:55:00,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1681844100672, max timestamp=1681844100677Num uniques 1024; 2023-04-18 18:55:00,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46894, new segment data size=10254, suffix heap size=134464,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1024, new segment cells count=224 2023-04-18 18:55:00,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1681844100672, max timestamp=1681844100677Num uniques 1024; 2023-04-18 18:55:00,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46894, new segment data size=10254, suffix heap size=134464,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1024, new segment cells count=224 2023-04-18 18:55:00,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=283, cellSize=12958, totalHeapSize=29672, min timestamp=1681844100689, max timestamp=1681844100698Num uniques 1083; 2023-04-18 18:55:00,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8378, totalHeapSize=19272, min timestamp=1681844100693, max timestamp=1681844100698Num uniques 1083; 2023-04-18 18:55:00,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=12958, suffix heap size=140600,new segment heap size=29672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=283 2023-04-18 18:55:00,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=8378, suffix heap size=140600,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=183 2023-04-18 18:55:00,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8378, totalHeapSize=19272, min timestamp=1681844100693, max timestamp=1681844100698Num uniques 1083; 2023-04-18 18:55:00,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=8378, suffix heap size=140600,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=183 2023-04-18 18:55:00,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=242, cellSize=11082, totalHeapSize=25408, min timestamp=1681844100714, max timestamp=1681844100719Num uniques 1142; 2023-04-18 18:55:00,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52302, new segment data size=11082, suffix heap size=146736,new segment heap size=25408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1142, new segment cells count=242 2023-04-18 18:55:00,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=242, cellSize=11082, totalHeapSize=25408, min timestamp=1681844100714, max timestamp=1681844100719Num uniques 1042; 2023-04-18 18:55:00,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=242, cellSize=11082, totalHeapSize=25408, min timestamp=1681844100714, max timestamp=1681844100719Num uniques 1042; 2023-04-18 18:55:00,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47722, new segment data size=11082, suffix heap size=136336,new segment heap size=25408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1042, new segment cells count=242 2023-04-18 18:55:00,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47722, new segment data size=11082, suffix heap size=136336,new segment heap size=25408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1042, new segment cells count=242 2023-04-18 18:55:00,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:00,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36732 deadline: 1681844160733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=151, cellSize=6915, totalHeapSize=15944, min timestamp=1681844100732, max timestamp=1681844100736Num uniques 1101; 2023-04-18 18:55:00,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50425, new segment data size=6915, suffix heap size=142472,new segment heap size=15944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1101, new segment cells count=151 2023-04-18 18:55:00,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=151, cellSize=6915, totalHeapSize=15944, min timestamp=1681844100732, max timestamp=1681844100736Num uniques 1101; 2023-04-18 18:55:00,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50425, new segment data size=6915, suffix heap size=142472,new segment heap size=15944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1101, new segment cells count=151 2023-04-18 18:55:00,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=151, cellSize=6915, totalHeapSize=15944, min timestamp=1681844100732, max timestamp=1681844100736Num uniques 1101; 2023-04-18 18:55:00,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50425, new segment data size=6915, suffix heap size=142472,new segment heap size=15944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1101, new segment cells count=151 2023-04-18 18:55:00,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=210, cellSize=9610, totalHeapSize=22080, min timestamp=1681844100743, max timestamp=1681844100752Num uniques 1010; 2023-04-18 18:55:00,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46250, new segment data size=9610, suffix heap size=133008,new segment heap size=22080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1010, new segment cells count=210 2023-04-18 18:55:00,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=210, cellSize=9610, totalHeapSize=22080, min timestamp=1681844100743, max timestamp=1681844100752Num uniques 1010; 2023-04-18 18:55:00,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=210, cellSize=9610, totalHeapSize=22080, min timestamp=1681844100743, max timestamp=1681844100752Num uniques 1010; 2023-04-18 18:55:00,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46250, new segment data size=9610, suffix heap size=133008,new segment heap size=22080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1010, new segment cells count=210 2023-04-18 18:55:00,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46250, new segment data size=9610, suffix heap size=133008,new segment heap size=22080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1010, new segment cells count=210 2023-04-18 18:55:00,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=269, cellSize=12314, totalHeapSize=28216, min timestamp=1681844100765, max timestamp=1681844100770Num uniques 1069; 2023-04-18 18:55:00,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48954, new segment data size=12314, suffix heap size=139144,new segment heap size=28216 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1069, new segment cells count=269 2023-04-18 18:55:00,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=269, cellSize=12314, totalHeapSize=28216, min timestamp=1681844100765, max timestamp=1681844100770Num uniques 1069; 2023-04-18 18:55:00,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48954, new segment data size=12314, suffix heap size=139144,new segment heap size=28216 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1069, new segment cells count=269 2023-04-18 18:55:00,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=269, cellSize=12314, totalHeapSize=28216, min timestamp=1681844100765, max timestamp=1681844100770Num uniques 1069; 2023-04-18 18:55:00,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48954, new segment data size=12314, suffix heap size=139144,new segment heap size=28216 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1069, new segment cells count=269 2023-04-18 18:55:00,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=228, cellSize=10438, totalHeapSize=23952, min timestamp=1681844100783, max timestamp=1681844100786Num uniques 1128; 2023-04-18 18:55:00,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51658, new segment data size=10438, suffix heap size=145280,new segment heap size=23952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1128, new segment cells count=228 2023-04-18 18:55:00,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:00,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36776 deadline: 1681844160785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=228, cellSize=10438, totalHeapSize=23952, min timestamp=1681844100783, max timestamp=1681844100786Num uniques 1128; 2023-04-18 18:55:00,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51658, new segment data size=10438, suffix heap size=145280,new segment heap size=23952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1128, new segment cells count=228 2023-04-18 18:55:00,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=228, cellSize=10438, totalHeapSize=23952, min timestamp=1681844100783, max timestamp=1681844100786Num uniques 1128; 2023-04-18 18:55:00,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51658, new segment data size=10438, suffix heap size=145280,new segment heap size=23952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1128, new segment cells count=228 2023-04-18 18:55:00,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=237, cellSize=10852, totalHeapSize=24888, min timestamp=1681844100799, max timestamp=1681844100807Num uniques 1087; 2023-04-18 18:55:00,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49782, new segment data size=10852, suffix heap size=141016,new segment heap size=24888 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1087, new segment cells count=237 2023-04-18 18:55:00,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=237, cellSize=10852, totalHeapSize=24888, min timestamp=1681844100799, max timestamp=1681844100807Num uniques 1087; 2023-04-18 18:55:00,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49782, new segment data size=10852, suffix heap size=141016,new segment heap size=24888 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1087, new segment cells count=237 2023-04-18 18:55:00,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=237, cellSize=10852, totalHeapSize=24888, min timestamp=1681844100799, max timestamp=1681844100807Num uniques 1087; 2023-04-18 18:55:00,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49782, new segment data size=10852, suffix heap size=141016,new segment heap size=24888 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1087, new segment cells count=237 2023-04-18 18:55:00,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=246, cellSize=11266, totalHeapSize=25824, min timestamp=1681844100818, max timestamp=1681844100825Num uniques 1096; 2023-04-18 18:55:00,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50196, new segment data size=11266, suffix heap size=141952,new segment heap size=25824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1096, new segment cells count=246 2023-04-18 18:55:00,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=246, cellSize=11266, totalHeapSize=25824, min timestamp=1681844100818, max timestamp=1681844100825Num uniques 1096; 2023-04-18 18:55:00,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50196, new segment data size=11266, suffix heap size=141952,new segment heap size=25824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1096, new segment cells count=246 2023-04-18 18:55:00,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=246, cellSize=11266, totalHeapSize=25824, min timestamp=1681844100818, max timestamp=1681844100825Num uniques 1096; 2023-04-18 18:55:00,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50196, new segment data size=11266, suffix heap size=141952,new segment heap size=25824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1096, new segment cells count=246 2023-04-18 18:55:00,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:00,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844160861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=205, cellSize=9385, totalHeapSize=21560, min timestamp=1681844100859, max timestamp=1681844100864Num uniques 1105; 2023-04-18 18:55:00,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=205, cellSize=9385, totalHeapSize=21560, min timestamp=1681844100859, max timestamp=1681844100864Num uniques 1105; 2023-04-18 18:55:00,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50605, new segment data size=9385, suffix heap size=142888,new segment heap size=21560 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1105, new segment cells count=205 2023-04-18 18:55:00,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50605, new segment data size=9385, suffix heap size=142888,new segment heap size=21560 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1105, new segment cells count=205 2023-04-18 18:55:00,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=205, cellSize=9385, totalHeapSize=21560, min timestamp=1681844100859, max timestamp=1681844100864Num uniques 1105; 2023-04-18 18:55:00,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50605, new segment data size=9385, suffix heap size=142888,new segment heap size=21560 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1105, new segment cells count=205 2023-04-18 18:55:00,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=164, cellSize=7504, totalHeapSize=17296, min timestamp=1681844100884, max timestamp=1681844100887Num uniques 1064; 2023-04-18 18:55:00,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48724, new segment data size=7504, suffix heap size=138624,new segment heap size=17296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1064, new segment cells count=164 2023-04-18 18:55:00,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=164, cellSize=7504, totalHeapSize=17296, min timestamp=1681844100884, max timestamp=1681844100887Num uniques 1064; 2023-04-18 18:55:00,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=164, cellSize=7504, totalHeapSize=17296, min timestamp=1681844100884, max timestamp=1681844100887Num uniques 1064; 2023-04-18 18:55:00,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48724, new segment data size=7504, suffix heap size=138624,new segment heap size=17296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1064, new segment cells count=164 2023-04-18 18:55:00,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48724, new segment data size=7504, suffix heap size=138624,new segment heap size=17296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1064, new segment cells count=164 2023-04-18 18:55:00,894 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=813 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/8d76954c6e60475cb94145334cf2cb0c 2023-04-18 18:55:00,899 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/02952e7825904e9a870504f4eba9bf03 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/02952e7825904e9a870504f4eba9bf03 2023-04-18 18:55:00,903 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/02952e7825904e9a870504f4eba9bf03, entries=100, sequenceid=813, filesize=9.5 K 2023-04-18 18:55:00,904 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/4f147476b46043eda5194cae866d3f48 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/4f147476b46043eda5194cae866d3f48 2023-04-18 18:55:00,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=223, cellSize=10208, totalHeapSize=23432, min timestamp=1681844100897, max timestamp=1681844100905Num uniques 1023; 2023-04-18 18:55:00,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=223, cellSize=10208, totalHeapSize=23432, min timestamp=1681844100897, max timestamp=1681844100905Num uniques 1023; 2023-04-18 18:55:00,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46848, new segment data size=10208, suffix heap size=134360,new segment heap size=23432 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1023, new segment cells count=223 2023-04-18 18:55:00,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=223, cellSize=10208, totalHeapSize=23432, min timestamp=1681844100897, max timestamp=1681844100905Num uniques 1023; 2023-04-18 18:55:00,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46848, new segment data size=10208, suffix heap size=134360,new segment heap size=23432 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1023, new segment cells count=223 2023-04-18 18:55:00,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46848, new segment data size=10208, suffix heap size=134360,new segment heap size=23432 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1023, new segment cells count=223 2023-04-18 18:55:00,911 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/4f147476b46043eda5194cae866d3f48, entries=100, sequenceid=813, filesize=9.5 K 2023-04-18 18:55:00,911 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/8d76954c6e60475cb94145334cf2cb0c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/8d76954c6e60475cb94145334cf2cb0c 2023-04-18 18:55:00,915 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/8d76954c6e60475cb94145334cf2cb0c, entries=100, sequenceid=813, filesize=9.5 K 2023-04-18 18:55:00,933 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=53.67 KB/54960 for e3c3c3367729f1159cdc97339f70376a in 1268ms, sequenceid=813, compaction requested=false 2023-04-18 18:55:00,933 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:55:00,933 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:55:00,934 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:00,934 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:00,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:55:00,934 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=73.80 KB heapSize=199.69 KB 2023-04-18 18:55:00,934 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,934 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:00,934 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:00,934 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:55:00,934 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:00,934 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:55:00,934 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:00,934 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:55:00,934 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:00,936 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:00,936 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:00,936 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:00,936 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:00,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:00,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21ec85c7[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:00,938 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:00,938 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:00,939 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:00,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:00,939 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,939 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:00,939 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:00,940 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:00,939 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:00,941 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:00,941 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,941 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:00,941 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:00,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:00,944 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:00,944 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:00,944 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:00,944 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:00,944 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:00,947 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:00,947 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:00,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:00,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:00,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:00,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:00,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:00,947 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:00,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:00,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:00,948 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:00,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:00,949 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:00,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:00,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,952 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:00,952 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:00,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:00,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:00,952 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,952 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:00,952 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:00,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:00,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:00,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:00,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:00,953 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:00,953 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:00,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:00,952 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:00,953 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,953 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:00,953 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:00,953 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:00,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:00,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36732 deadline: 1681844160955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:00,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 616 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844160956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:00,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36776 deadline: 1681844160956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:00,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 536 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844160957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,964 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1631 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/1dca17440c8c4eed8f616cbf35a445b0 2023-04-18 18:55:00,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:00,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844160966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:00,975 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1631 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/565e6aa48d9741efbe552bc4980b2c53 2023-04-18 18:55:01,053 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:01,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:01,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:01,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:01,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:01,056 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:01,056 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:01,056 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:01,056 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:01,057 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,057 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:01,058 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:01,058 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:01,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:01,058 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,058 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:01,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:01,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:01,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:01,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:01,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:01,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:01,060 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:01,060 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:01,060 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:01,060 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:01,060 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:01,060 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:55:01,060 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:55:01,060 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:55:01,060 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:55:01,060 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:01,060 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:01,060 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,060 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:01,060 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:01,062 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,062 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:01,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:01,062 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:01,062 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:01,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:01,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:01,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:01,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 618 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844161060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36732 deadline: 1681844161059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:01,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36776 deadline: 1681844161060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:01,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:01,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@266dcb38[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:01,064 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:01,064 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:01,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 538 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844161063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,066 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:01,066 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,066 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:01,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:01,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:01,067 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,067 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:01,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:01,068 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:01,068 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:01,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:01,070 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:01,070 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:01,070 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:01,070 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:01,070 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:01,071 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,071 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,072 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:01,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:01,072 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:01,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,074 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,075 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,076 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:01,076 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:01,076 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:01,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:01,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:01,076 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:01,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:01,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:01,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:01,076 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:01,076 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:01,076 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,076 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,076 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,076 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,076 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:01,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:01,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:01,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:01,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844161169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,176 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:01,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:01,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:01,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:01,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:01,179 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:01,179 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:01,179 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:01,179 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:01,180 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,180 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:01,182 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,182 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:01,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:01,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:01,182 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,182 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:01,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:01,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:01,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:01,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:01,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:01,183 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:01,183 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:01,183 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:01,183 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:01,183 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:01,183 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:55:01,184 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:55:01,184 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:55:01,184 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:55:01,184 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:01,184 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:01,184 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,184 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:01,184 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:01,185 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,185 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:01,185 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,185 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:01,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:01,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:01,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:01,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32ef8f25[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:01,187 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:01,187 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,188 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,188 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:01,188 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:01,188 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:01,188 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,188 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:01,189 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,189 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:01,189 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:01,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:01,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:01,189 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:01,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:01,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:01,192 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:01,192 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:01,192 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:01,192 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:01,192 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:01,195 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,195 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:01,195 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:01,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,196 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:01,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:01,197 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:01,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,203 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:01,203 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:01,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:01,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:01,203 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:01,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:01,203 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:01,203 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:01,203 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:01,203 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:01,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:01,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:01,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:01,203 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:01,203 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,204 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,204 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,204 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:01,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36776 deadline: 1681844161264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:01,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36732 deadline: 1681844161265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:01,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 620 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844161265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:01,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 540 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844161267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,304 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:01,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:01,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:01,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:01,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:01,307 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:01,307 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:01,307 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:01,307 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:01,307 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,308 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:01,309 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,309 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:01,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:01,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:01,309 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,309 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:01,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:01,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:01,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:01,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:01,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:01,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:01,310 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:01,310 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:01,311 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:01,311 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:01,311 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:01,311 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:55:01,311 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:55:01,311 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:55:01,311 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:55:01,311 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:01,311 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:01,311 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,311 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:01,311 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:01,314 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,314 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:01,314 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,314 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:01,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:01,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:01,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:01,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33e06dc6[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:01,316 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:01,316 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,317 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,317 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:01,317 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:01,317 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:01,317 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,317 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:01,318 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:01,318 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:01,318 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,318 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:01,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:01,321 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:01,321 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:01,321 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:01,321 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:01,321 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:01,322 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,322 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,322 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:01,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:01,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:01,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:01,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:01,323 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:01,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:01,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,324 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:01,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,327 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,327 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:01,327 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:01,327 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:01,327 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:01,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:01,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:01,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:01,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:01,328 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,328 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:01,328 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:01,328 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:01,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:01,328 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:01,328 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:01,328 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,385 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1631 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/3c7a7d9689d74f878ac3d2fa27889eb5 2023-04-18 18:55:01,389 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/1dca17440c8c4eed8f616cbf35a445b0 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/1dca17440c8c4eed8f616cbf35a445b0 2023-04-18 18:55:01,393 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/1dca17440c8c4eed8f616cbf35a445b0, entries=150, sequenceid=1631, filesize=11.9 K 2023-04-18 18:55:01,394 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/565e6aa48d9741efbe552bc4980b2c53 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/565e6aa48d9741efbe552bc4980b2c53 2023-04-18 18:55:01,397 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/565e6aa48d9741efbe552bc4980b2c53, entries=150, sequenceid=1631, filesize=11.9 K 2023-04-18 18:55:01,398 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/3c7a7d9689d74f878ac3d2fa27889eb5 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/3c7a7d9689d74f878ac3d2fa27889eb5 2023-04-18 18:55:01,401 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/3c7a7d9689d74f878ac3d2fa27889eb5, entries=150, sequenceid=1631, filesize=11.9 K 2023-04-18 18:55:01,402 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~199.64 KB/204432, currentSize=107.34 KB/109920 for e3c3c3367729f1159cdc97339f70376a in 468ms, sequenceid=1631, compaction requested=true 2023-04-18 18:55:01,402 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:55:01,402 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:01,402 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:01,402 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:01,402 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:01,402 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:01,403 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:01,403 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:01,403 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/A is initiating minor compaction (all files) 2023-04-18 18:55:01,403 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/B is initiating minor compaction (all files) 2023-04-18 18:55:01,403 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/A in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:55:01,403 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/B in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:55:01,403 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/98ef8efecb4d4a7596d9b1de23a4c209, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/02952e7825904e9a870504f4eba9bf03, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/1dca17440c8c4eed8f616cbf35a445b0] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=34.2 K 2023-04-18 18:55:01,403 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/0a4cd85f63a343cfb1f6d703a302a032, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/4f147476b46043eda5194cae866d3f48, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/565e6aa48d9741efbe552bc4980b2c53] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=34.2 K 2023-04-18 18:55:01,404 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 98ef8efecb4d4a7596d9b1de23a4c209, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=804, earliestPutTs=1681844098461 2023-04-18 18:55:01,404 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 0a4cd85f63a343cfb1f6d703a302a032, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=804, earliestPutTs=1681844098461 2023-04-18 18:55:01,404 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 02952e7825904e9a870504f4eba9bf03, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=813, earliestPutTs=1681844099090 2023-04-18 18:55:01,404 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 4f147476b46043eda5194cae866d3f48, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=813, earliestPutTs=1681844099090 2023-04-18 18:55:01,404 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 1dca17440c8c4eed8f616cbf35a445b0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1631, earliestPutTs=1681844100909 2023-04-18 18:55:01,405 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 565e6aa48d9741efbe552bc4980b2c53, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1631, earliestPutTs=1681844100909 2023-04-18 18:55:01,411 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#B#compaction#288 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:01,411 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#A#compaction#287 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:01,428 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:01,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:01,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:01,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:01,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:01,432 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:01,432 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:01,432 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:01,432 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:01,433 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,433 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:01,434 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,434 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:01,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:01,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:01,434 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,434 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:01,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:01,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:01,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:01,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:01,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:01,436 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:01,436 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:01,436 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:55:01,436 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. started... 2023-04-18 18:55:01,436 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-18 18:55:01,436 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:55:01,436 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:01,436 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:55:01,436 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:01,436 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:55:01,436 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:01,446 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1650 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/c3e9516de80a4f129f76436b8dd398ad 2023-04-18 18:55:01,457 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1650 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/d345e079bfad4d7da8090ccdd6d3d738 2023-04-18 18:55:01,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:55:01,477 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. as already flushing 2023-04-18 18:55:01,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:01,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844161501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:01,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36776 deadline: 1681844161565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:01,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36732 deadline: 1681844161569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:01,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 542 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36688 deadline: 1681844161570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:01,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 622 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36772 deadline: 1681844161570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:01,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844161605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:01,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36696 deadline: 1681844161808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,821 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/c031218c82af486bb69525129fab1ade as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/c031218c82af486bb69525129fab1ade 2023-04-18 18:55:01,822 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/2c3565896f3646eda74b13dbc671ce5a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/2c3565896f3646eda74b13dbc671ce5a 2023-04-18 18:55:01,825 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/A of e3c3c3367729f1159cdc97339f70376a into c031218c82af486bb69525129fab1ade(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:01,826 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:55:01,826 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/A, priority=13, startTime=1681844101402; duration=0sec 2023-04-18 18:55:01,826 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:01,826 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:01,827 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:01,827 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): e3c3c3367729f1159cdc97339f70376a/C is initiating minor compaction (all files) 2023-04-18 18:55:01,827 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e3c3c3367729f1159cdc97339f70376a/C in TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:55:01,827 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/5f5851c62aa64aec8bd1ce9400719acc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/8d76954c6e60475cb94145334cf2cb0c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/3c7a7d9689d74f878ac3d2fa27889eb5] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp, totalSize=34.2 K 2023-04-18 18:55:01,827 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 5f5851c62aa64aec8bd1ce9400719acc, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=804, earliestPutTs=1681844098461 2023-04-18 18:55:01,828 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/B of e3c3c3367729f1159cdc97339f70376a into 2c3565896f3646eda74b13dbc671ce5a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:01,828 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 8d76954c6e60475cb94145334cf2cb0c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=813, earliestPutTs=1681844099090 2023-04-18 18:55:01,828 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:55:01,828 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/B, priority=13, startTime=1681844101402; duration=0sec 2023-04-18 18:55:01,828 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:01,828 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 3c7a7d9689d74f878ac3d2fa27889eb5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1631, earliestPutTs=1681844100909 2023-04-18 18:55:01,834 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): e3c3c3367729f1159cdc97339f70376a#C#compaction#292 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:01,866 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/b309b8970e7142149301061cd022f691 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b309b8970e7142149301061cd022f691 2023-04-18 18:55:01,870 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3c3c3367729f1159cdc97339f70376a/C of e3c3c3367729f1159cdc97339f70376a into b309b8970e7142149301061cd022f691(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:01,870 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:55:01,870 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a., storeName=e3c3c3367729f1159cdc97339f70376a/C, priority=13, startTime=1681844101402; duration=0sec 2023-04-18 18:55:01,871 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:01,877 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1650 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/a6c035d4c2974f5c93d759524b64f067 2023-04-18 18:55:01,882 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/c3e9516de80a4f129f76436b8dd398ad as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/c3e9516de80a4f129f76436b8dd398ad 2023-04-18 18:55:01,885 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/c3e9516de80a4f129f76436b8dd398ad, entries=150, sequenceid=1650, filesize=11.9 K 2023-04-18 18:55:01,886 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/d345e079bfad4d7da8090ccdd6d3d738 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/d345e079bfad4d7da8090ccdd6d3d738 2023-04-18 18:55:01,889 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/d345e079bfad4d7da8090ccdd6d3d738, entries=150, sequenceid=1650, filesize=11.9 K 2023-04-18 18:55:01,890 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/a6c035d4c2974f5c93d759524b64f067 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/a6c035d4c2974f5c93d759524b64f067 2023-04-18 18:55:01,894 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/a6c035d4c2974f5c93d759524b64f067, entries=150, sequenceid=1650, filesize=11.9 K 2023-04-18 18:55:01,894 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for e3c3c3367729f1159cdc97339f70376a in 458ms, sequenceid=1650, compaction requested=false 2023-04-18 18:55:01,894 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:55:01,894 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:55:01,894 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:01,895 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:01,895 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,895 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:01,895 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:01,896 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:01,896 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:01,897 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,897 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:01,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:01,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@766704c0[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:01,898 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:01,898 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,899 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,900 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:01,900 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:01,900 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:01,900 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,900 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:01,902 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,902 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:01,902 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:01,902 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:01,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:01,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:01,905 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:01,905 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:01,905 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:01,905 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:01,905 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:01,907 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,907 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:01,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,907 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:01,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:01,907 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:01,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,909 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:01,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:01,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,912 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,912 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:01,912 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:01,912 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:01,912 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:01,912 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:01,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:01,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:01,912 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:01,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:01,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:01,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:01,912 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:01,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:01,913 DEBUG [Thread-1526] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:01,913 DEBUG [Thread-1526] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:01,913 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:01,913 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:01,955 DEBUG [Thread-1531] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69bc154a to 127.0.0.1:65091 2023-04-18 18:55:01,955 DEBUG [Thread-1531] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:01,956 DEBUG [Thread-1535] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13e90e8d to 127.0.0.1:65091 2023-04-18 18:55:01,956 DEBUG [Thread-1527] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x574000bf to 127.0.0.1:65091 2023-04-18 18:55:01,956 DEBUG [Thread-1535] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:01,956 DEBUG [Thread-1527] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:01,956 DEBUG [Thread-1533] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15186778 to 127.0.0.1:65091 2023-04-18 18:55:01,956 DEBUG [Thread-1533] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:01,957 DEBUG [Thread-1529] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4994df89 to 127.0.0.1:65091 2023-04-18 18:55:01,957 DEBUG [Thread-1529] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:02,013 DEBUG [Thread-1526] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:02,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:02,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:55:02,070 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-18 18:55:02,070 DEBUG [Thread-1520] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x76203061 to 127.0.0.1:65091 2023-04-18 18:55:02,070 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:55:02,070 DEBUG [Thread-1520] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:02,070 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:02,070 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:55:02,070 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:02,070 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:55:02,070 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:02,073 DEBUG [Thread-1516] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5096c69b to 127.0.0.1:65091 2023-04-18 18:55:02,073 DEBUG [Thread-1516] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:02,075 DEBUG [Thread-1522] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b39d5b8 to 127.0.0.1:65091 2023-04-18 18:55:02,075 DEBUG [Thread-1522] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:02,076 DEBUG [Thread-1518] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x216aad74 to 127.0.0.1:65091 2023-04-18 18:55:02,076 DEBUG [Thread-1518] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:02,112 DEBUG [Thread-1524] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x44f9bd75 to 127.0.0.1:65091 2023-04-18 18:55:02,112 DEBUG [Thread-1524] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:02,113 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-18 18:55:02,113 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 425 2023-04-18 18:55:02,113 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 499 2023-04-18 18:55:02,113 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 155 2023-04-18 18:55:02,113 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 186 2023-04-18 18:55:02,113 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 303 2023-04-18 18:55:02,113 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-18 18:55:02,113 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(398): read 5126 2023-04-18 18:55:02,113 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(398): read 5085 2023-04-18 18:55:02,113 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(398): read 5127 2023-04-18 18:55:02,113 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(398): read 5124 2023-04-18 18:55:02,113 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(398): read 5075 2023-04-18 18:55:02,113 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-18 18:55:02,113 INFO [Listener at localhost/36951] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-18 18:55:02,113 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d7cdc2b to 127.0.0.1:65091 2023-04-18 18:55:02,113 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:02,114 INFO [Listener at localhost/36951] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-18 18:55:02,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-18 18:55:02,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-18 18:55:02,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-18 18:55:02,117 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844102117"}]},"ts":"1681844102117"} 2023-04-18 18:55:02,118 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-18 18:55:02,120 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-18 18:55:02,120 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e3c3c3367729f1159cdc97339f70376a, UNASSIGN}] 2023-04-18 18:55:02,122 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e3c3c3367729f1159cdc97339f70376a, UNASSIGN 2023-04-18 18:55:02,122 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=e3c3c3367729f1159cdc97339f70376a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:02,122 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844102122"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681844102122"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681844102122"}]},"ts":"1681844102122"} 2023-04-18 18:55:02,123 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952}] 2023-04-18 18:55:02,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-18 18:55:02,275 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:55:02,275 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e3c3c3367729f1159cdc97339f70376a, disabling compactions & flushes 2023-04-18 18:55:02,275 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:55:02,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-18 18:55:02,477 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1667 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/f2ba448e302a40618dd8b5e048ecc125 2023-04-18 18:55:02,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-18 18:55:02,886 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1667 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/49d88fa6315f4f339f560819698485d1 2023-04-18 18:55:03,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-18 18:55:03,294 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1667 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/45fb9b1d064546ca87451b5a5e673fba 2023-04-18 18:55:03,298 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/f2ba448e302a40618dd8b5e048ecc125 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/f2ba448e302a40618dd8b5e048ecc125 2023-04-18 18:55:03,301 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/f2ba448e302a40618dd8b5e048ecc125, entries=150, sequenceid=1667, filesize=11.9 K 2023-04-18 18:55:03,302 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/49d88fa6315f4f339f560819698485d1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/49d88fa6315f4f339f560819698485d1 2023-04-18 18:55:03,305 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/49d88fa6315f4f339f560819698485d1, entries=150, sequenceid=1667, filesize=11.9 K 2023-04-18 18:55:03,306 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/45fb9b1d064546ca87451b5a5e673fba as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/45fb9b1d064546ca87451b5a5e673fba 2023-04-18 18:55:03,309 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/45fb9b1d064546ca87451b5a5e673fba, entries=150, sequenceid=1667, filesize=11.9 K 2023-04-18 18:55:03,309 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=26.84 KB/27480 for e3c3c3367729f1159cdc97339f70376a in 1239ms, sequenceid=1667, compaction requested=true 2023-04-18 18:55:03,309 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:55:03,310 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:55:03,310 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. because compaction request was cancelled 2023-04-18 18:55:03,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:55:03,310 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:03,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. after waiting 0 ms 2023-04-18 18:55:03,310 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. because compaction request was cancelled 2023-04-18 18:55:03,310 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:03,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:55:03,310 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:03,310 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing e3c3c3367729f1159cdc97339f70376a 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-18 18:55:03,310 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. because compaction request was cancelled 2023-04-18 18:55:03,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=A 2023-04-18 18:55:03,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:03,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=B 2023-04-18 18:55:03,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:03,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3c3c3367729f1159cdc97339f70376a, store=C 2023-04-18 18:55:03,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:03,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1674 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/496e09bb198e460389981c2d9c54ff48 2023-04-18 18:55:04,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1674 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/d2663579d9d74c2bbf2fd96577048310 2023-04-18 18:55:04,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-18 18:55:04,536 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1674 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/ba490ea8d1644eaaaadd112293526816 2023-04-18 18:55:04,539 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/A/496e09bb198e460389981c2d9c54ff48 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/496e09bb198e460389981c2d9c54ff48 2023-04-18 18:55:04,543 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/496e09bb198e460389981c2d9c54ff48, entries=100, sequenceid=1674, filesize=9.5 K 2023-04-18 18:55:04,543 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/B/d2663579d9d74c2bbf2fd96577048310 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/d2663579d9d74c2bbf2fd96577048310 2023-04-18 18:55:04,547 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/d2663579d9d74c2bbf2fd96577048310, entries=100, sequenceid=1674, filesize=9.5 K 2023-04-18 18:55:04,547 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/.tmp/C/ba490ea8d1644eaaaadd112293526816 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/ba490ea8d1644eaaaadd112293526816 2023-04-18 18:55:04,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/ba490ea8d1644eaaaadd112293526816, entries=100, sequenceid=1674, filesize=9.5 K 2023-04-18 18:55:04,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for e3c3c3367729f1159cdc97339f70376a in 1241ms, sequenceid=1674, compaction requested=true 2023-04-18 18:55:04,574 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/ded014cc35fa4880ada9724ed3b5158f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/4a295d84e69f4ba09f035db3fe13c51e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/5a52b1f63e0e4b27a96dca32ba575b9e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/f489bbb9a94e4bd68632003dd3941b3b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/21a631e04ad24281acefe3af5b71ce97, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/5a08f7d605a34cb28e31d7eb44fe7d57, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/807a8db805954a9a915237af046ba6a8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/1318b43be7a2491ea10792c6f55ed019, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/2ecb14c5e57c4b1e8c0fdb6a2c141895, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/fc4ac4a8606345a7807c569620a4e4d3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/d2234835849b44058ce2cd232d9d1eea, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/688920c1b0e34d9b96b7ba31335ec574, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/5568ef5003844af8b48ba0dc5c979ba8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/76cf7abf00534c4e87ead50f1ec4ca49, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/2ff56496abca4374bc313d04c42365e1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/3a1caf5c45e8436aaf444a09ede78e36, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/0c059abb2b964d91a6540ca35f09667e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/d1360d5d68a94910940a61f4ad6f7b75, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/936ad3a4f9d24b4fa524b19b3411cb31, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/03c8473a119d4d1c970867b7270973c3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/608b622b39c44a559520cb0a13a99774, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/2193ed5db95f4ac88c3509f5540ce682, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/f85c8bc026d64a7f827731993159122b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/ae9f3e7edc2c4b2d837828a7422892c1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/eca997c5eb374839bdd3c28c6f99349c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/09f6c84a01254a3db7303e40fef481d9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/40259220b7a641238ee19f2f74b78d9b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/98ef8efecb4d4a7596d9b1de23a4c209, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/ebb99bc4c27d474a9766adea3723547b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/02952e7825904e9a870504f4eba9bf03, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/1dca17440c8c4eed8f616cbf35a445b0] to archive 2023-04-18 18:55:04,575 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 18:55:04,576 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/ded014cc35fa4880ada9724ed3b5158f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/ded014cc35fa4880ada9724ed3b5158f 2023-04-18 18:55:04,577 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/4a295d84e69f4ba09f035db3fe13c51e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/4a295d84e69f4ba09f035db3fe13c51e 2023-04-18 18:55:04,578 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/5a52b1f63e0e4b27a96dca32ba575b9e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/5a52b1f63e0e4b27a96dca32ba575b9e 2023-04-18 18:55:04,579 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/f489bbb9a94e4bd68632003dd3941b3b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/f489bbb9a94e4bd68632003dd3941b3b 2023-04-18 18:55:04,580 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/21a631e04ad24281acefe3af5b71ce97 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/21a631e04ad24281acefe3af5b71ce97 2023-04-18 18:55:04,581 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/5a08f7d605a34cb28e31d7eb44fe7d57 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/5a08f7d605a34cb28e31d7eb44fe7d57 2023-04-18 18:55:04,582 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/807a8db805954a9a915237af046ba6a8 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/807a8db805954a9a915237af046ba6a8 2023-04-18 18:55:04,583 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/1318b43be7a2491ea10792c6f55ed019 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/1318b43be7a2491ea10792c6f55ed019 2023-04-18 18:55:04,584 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/2ecb14c5e57c4b1e8c0fdb6a2c141895 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/2ecb14c5e57c4b1e8c0fdb6a2c141895 2023-04-18 18:55:04,585 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/fc4ac4a8606345a7807c569620a4e4d3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/fc4ac4a8606345a7807c569620a4e4d3 2023-04-18 18:55:04,586 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/d2234835849b44058ce2cd232d9d1eea to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/d2234835849b44058ce2cd232d9d1eea 2023-04-18 18:55:04,587 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/688920c1b0e34d9b96b7ba31335ec574 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/688920c1b0e34d9b96b7ba31335ec574 2023-04-18 18:55:04,587 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/5568ef5003844af8b48ba0dc5c979ba8 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/5568ef5003844af8b48ba0dc5c979ba8 2023-04-18 18:55:04,588 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/76cf7abf00534c4e87ead50f1ec4ca49 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/76cf7abf00534c4e87ead50f1ec4ca49 2023-04-18 18:55:04,589 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/2ff56496abca4374bc313d04c42365e1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/2ff56496abca4374bc313d04c42365e1 2023-04-18 18:55:04,590 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/3a1caf5c45e8436aaf444a09ede78e36 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/3a1caf5c45e8436aaf444a09ede78e36 2023-04-18 18:55:04,591 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/0c059abb2b964d91a6540ca35f09667e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/0c059abb2b964d91a6540ca35f09667e 2023-04-18 18:55:04,592 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/d1360d5d68a94910940a61f4ad6f7b75 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/d1360d5d68a94910940a61f4ad6f7b75 2023-04-18 18:55:04,593 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/936ad3a4f9d24b4fa524b19b3411cb31 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/936ad3a4f9d24b4fa524b19b3411cb31 2023-04-18 18:55:04,594 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/03c8473a119d4d1c970867b7270973c3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/03c8473a119d4d1c970867b7270973c3 2023-04-18 18:55:04,594 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/608b622b39c44a559520cb0a13a99774 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/608b622b39c44a559520cb0a13a99774 2023-04-18 18:55:04,595 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/2193ed5db95f4ac88c3509f5540ce682 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/2193ed5db95f4ac88c3509f5540ce682 2023-04-18 18:55:04,596 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/f85c8bc026d64a7f827731993159122b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/f85c8bc026d64a7f827731993159122b 2023-04-18 18:55:04,597 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/ae9f3e7edc2c4b2d837828a7422892c1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/ae9f3e7edc2c4b2d837828a7422892c1 2023-04-18 18:55:04,598 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/eca997c5eb374839bdd3c28c6f99349c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/eca997c5eb374839bdd3c28c6f99349c 2023-04-18 18:55:04,598 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/09f6c84a01254a3db7303e40fef481d9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/09f6c84a01254a3db7303e40fef481d9 2023-04-18 18:55:04,599 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/40259220b7a641238ee19f2f74b78d9b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/40259220b7a641238ee19f2f74b78d9b 2023-04-18 18:55:04,600 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/98ef8efecb4d4a7596d9b1de23a4c209 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/98ef8efecb4d4a7596d9b1de23a4c209 2023-04-18 18:55:04,601 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/ebb99bc4c27d474a9766adea3723547b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/ebb99bc4c27d474a9766adea3723547b 2023-04-18 18:55:04,602 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/02952e7825904e9a870504f4eba9bf03 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/02952e7825904e9a870504f4eba9bf03 2023-04-18 18:55:04,603 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/1dca17440c8c4eed8f616cbf35a445b0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/1dca17440c8c4eed8f616cbf35a445b0 2023-04-18 18:55:04,606 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/60de32464a4545e6945720fb7fcfc843, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/0a230ea48e6b417db6923b93090721c8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/8c786d850e68420f9c80cb61b5a8b2ae, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/f7bb2dd03b364ccca404cccdf5af21b5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/95523899589f420fa2b42d157513d30a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/1db7b88f1c2f42fbac872559507c391e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/9f7f9782511e492a95c0463ffc111d21, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/60b86b29438241869d747a4bced1021b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/d234e63ab9f64a3e828b9dc625246acf, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/53a8ec21aab24103b3997131abc7ff12, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/79185b69e0d0469f880c2cc0ee48f973, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/ea5e0d4c67054e538e5107a94f4f4fd4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/c564c2fe4eed43e8a5ffd80fc22b37dc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/564b14ca32f149b5b21e61d216ecd781, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/b8a46de2595a4b0f89e97c3d82d11cb1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/576bacdabcbe43ad858a6733b962f61f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/788cfaf5966b432b89987015941589cb, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/1bcc18ce547c499fac587737d7482f4c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/da10ad07a7c3424bb8d5f4633c5ff7be, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/743f5565ef7d40549fd364bf130b9d78, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/73165cfc62e0405b92d2c64dc21ace3c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/e668d2466fd04cab89ad19ddf779d7d8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/86c26ddd1281424f9abb50cde3db3d12, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/66b5bcbf9538491a967503a5ae61e9f0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/fe6e3c82871a4f9699ed2db672aa33ec, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/4e5fe921c8a24927bcaf9a818b59fae1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/35bc2754f92f450abe915a8fb047c0e6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/0a4cd85f63a343cfb1f6d703a302a032, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/849b0310f4a24753a4102462ae7775b1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/4f147476b46043eda5194cae866d3f48, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/565e6aa48d9741efbe552bc4980b2c53] to archive 2023-04-18 18:55:04,606 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 18:55:04,608 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/60de32464a4545e6945720fb7fcfc843 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/60de32464a4545e6945720fb7fcfc843 2023-04-18 18:55:04,608 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/0a230ea48e6b417db6923b93090721c8 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/0a230ea48e6b417db6923b93090721c8 2023-04-18 18:55:04,609 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/8c786d850e68420f9c80cb61b5a8b2ae to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/8c786d850e68420f9c80cb61b5a8b2ae 2023-04-18 18:55:04,610 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/f7bb2dd03b364ccca404cccdf5af21b5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/f7bb2dd03b364ccca404cccdf5af21b5 2023-04-18 18:55:04,611 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/95523899589f420fa2b42d157513d30a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/95523899589f420fa2b42d157513d30a 2023-04-18 18:55:04,612 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/1db7b88f1c2f42fbac872559507c391e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/1db7b88f1c2f42fbac872559507c391e 2023-04-18 18:55:04,613 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/9f7f9782511e492a95c0463ffc111d21 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/9f7f9782511e492a95c0463ffc111d21 2023-04-18 18:55:04,614 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/60b86b29438241869d747a4bced1021b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/60b86b29438241869d747a4bced1021b 2023-04-18 18:55:04,615 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/d234e63ab9f64a3e828b9dc625246acf to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/d234e63ab9f64a3e828b9dc625246acf 2023-04-18 18:55:04,616 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/53a8ec21aab24103b3997131abc7ff12 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/53a8ec21aab24103b3997131abc7ff12 2023-04-18 18:55:04,617 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/79185b69e0d0469f880c2cc0ee48f973 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/79185b69e0d0469f880c2cc0ee48f973 2023-04-18 18:55:04,618 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/ea5e0d4c67054e538e5107a94f4f4fd4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/ea5e0d4c67054e538e5107a94f4f4fd4 2023-04-18 18:55:04,618 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/c564c2fe4eed43e8a5ffd80fc22b37dc to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/c564c2fe4eed43e8a5ffd80fc22b37dc 2023-04-18 18:55:04,619 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/564b14ca32f149b5b21e61d216ecd781 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/564b14ca32f149b5b21e61d216ecd781 2023-04-18 18:55:04,620 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/b8a46de2595a4b0f89e97c3d82d11cb1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/b8a46de2595a4b0f89e97c3d82d11cb1 2023-04-18 18:55:04,621 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/576bacdabcbe43ad858a6733b962f61f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/576bacdabcbe43ad858a6733b962f61f 2023-04-18 18:55:04,622 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/788cfaf5966b432b89987015941589cb to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/788cfaf5966b432b89987015941589cb 2023-04-18 18:55:04,623 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/1bcc18ce547c499fac587737d7482f4c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/1bcc18ce547c499fac587737d7482f4c 2023-04-18 18:55:04,623 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/da10ad07a7c3424bb8d5f4633c5ff7be to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/da10ad07a7c3424bb8d5f4633c5ff7be 2023-04-18 18:55:04,624 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/743f5565ef7d40549fd364bf130b9d78 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/743f5565ef7d40549fd364bf130b9d78 2023-04-18 18:55:04,625 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/73165cfc62e0405b92d2c64dc21ace3c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/73165cfc62e0405b92d2c64dc21ace3c 2023-04-18 18:55:04,626 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/e668d2466fd04cab89ad19ddf779d7d8 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/e668d2466fd04cab89ad19ddf779d7d8 2023-04-18 18:55:04,627 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/86c26ddd1281424f9abb50cde3db3d12 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/86c26ddd1281424f9abb50cde3db3d12 2023-04-18 18:55:04,628 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/66b5bcbf9538491a967503a5ae61e9f0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/66b5bcbf9538491a967503a5ae61e9f0 2023-04-18 18:55:04,629 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/fe6e3c82871a4f9699ed2db672aa33ec to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/fe6e3c82871a4f9699ed2db672aa33ec 2023-04-18 18:55:04,629 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/4e5fe921c8a24927bcaf9a818b59fae1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/4e5fe921c8a24927bcaf9a818b59fae1 2023-04-18 18:55:04,630 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/35bc2754f92f450abe915a8fb047c0e6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/35bc2754f92f450abe915a8fb047c0e6 2023-04-18 18:55:04,631 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/0a4cd85f63a343cfb1f6d703a302a032 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/0a4cd85f63a343cfb1f6d703a302a032 2023-04-18 18:55:04,632 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/849b0310f4a24753a4102462ae7775b1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/849b0310f4a24753a4102462ae7775b1 2023-04-18 18:55:04,633 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/4f147476b46043eda5194cae866d3f48 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/4f147476b46043eda5194cae866d3f48 2023-04-18 18:55:04,633 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/565e6aa48d9741efbe552bc4980b2c53 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/565e6aa48d9741efbe552bc4980b2c53 2023-04-18 18:55:04,636 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/6cae18f511504c9aa01e981e241d7186, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/fd89a62a7bfa4b758ef02e83f286009d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b0917fd341184707be47c165186394df, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/4b035617edec47b69a58cb13893f317c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/f9c75fe9139e4fef8f8ed687cb8eb917, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/5ae29c80b457462ab7ce0c66a5a1595a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bc24fab781c946828b93b0710f5966a2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bab22ccb86a44aff8781e5edef1d38f7, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/8d0eba14a5e347969fb0889173a736e2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b7c00c149cd843cd9eb39deb4fc1c00e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bdf24e6770cd4ddab49945acfc54b505, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/c5aac3f5231e4248bb91b9619025bc75, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/0c9dfb27713d437f8494e5a4f1937880, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/4ba2c3f7c3d74f9bb9a5c0f08091acb7, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bebd2dd5b10c4498a3f9763b643289c2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b355a132009f4d199985a25eb4b6abf1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/2a14f50354f741c1b59bfb04a8e02558, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/c8f83b60c70b4a7dbf22ad6768851bcc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/c801ebe22f6849648b750b70c0d3b068, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/041e601490a7465ba23943a14250da5c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b0334c76c3c5439499b61b051d44dae3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/afc8eb42667c410797456e9e1c56a05d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/4a3781aa449a4bcaa1fc27c6db83f0db, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/a9aa45f6aa0947528c1363c2ef98545e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/e73c2c866c6040bbbb6d1204224ef1d6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/73d0655bd8f949699d4e6eb0030b3434, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/a1fba916d7cd4e1eaca275147f7133dc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/5f5851c62aa64aec8bd1ce9400719acc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/be08611e9fb64373a733fdd0e40b1f51, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/8d76954c6e60475cb94145334cf2cb0c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/3c7a7d9689d74f878ac3d2fa27889eb5] to archive 2023-04-18 18:55:04,637 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 18:55:04,638 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/6cae18f511504c9aa01e981e241d7186 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/6cae18f511504c9aa01e981e241d7186 2023-04-18 18:55:04,639 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/fd89a62a7bfa4b758ef02e83f286009d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/fd89a62a7bfa4b758ef02e83f286009d 2023-04-18 18:55:04,640 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b0917fd341184707be47c165186394df to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b0917fd341184707be47c165186394df 2023-04-18 18:55:04,641 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/4b035617edec47b69a58cb13893f317c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/4b035617edec47b69a58cb13893f317c 2023-04-18 18:55:04,641 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/f9c75fe9139e4fef8f8ed687cb8eb917 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/f9c75fe9139e4fef8f8ed687cb8eb917 2023-04-18 18:55:04,642 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/5ae29c80b457462ab7ce0c66a5a1595a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/5ae29c80b457462ab7ce0c66a5a1595a 2023-04-18 18:55:04,643 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bc24fab781c946828b93b0710f5966a2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bc24fab781c946828b93b0710f5966a2 2023-04-18 18:55:04,644 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bab22ccb86a44aff8781e5edef1d38f7 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bab22ccb86a44aff8781e5edef1d38f7 2023-04-18 18:55:04,646 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/8d0eba14a5e347969fb0889173a736e2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/8d0eba14a5e347969fb0889173a736e2 2023-04-18 18:55:04,647 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b7c00c149cd843cd9eb39deb4fc1c00e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b7c00c149cd843cd9eb39deb4fc1c00e 2023-04-18 18:55:04,648 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bdf24e6770cd4ddab49945acfc54b505 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bdf24e6770cd4ddab49945acfc54b505 2023-04-18 18:55:04,649 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/c5aac3f5231e4248bb91b9619025bc75 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/c5aac3f5231e4248bb91b9619025bc75 2023-04-18 18:55:04,650 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/0c9dfb27713d437f8494e5a4f1937880 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/0c9dfb27713d437f8494e5a4f1937880 2023-04-18 18:55:04,652 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/4ba2c3f7c3d74f9bb9a5c0f08091acb7 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/4ba2c3f7c3d74f9bb9a5c0f08091acb7 2023-04-18 18:55:04,653 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bebd2dd5b10c4498a3f9763b643289c2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/bebd2dd5b10c4498a3f9763b643289c2 2023-04-18 18:55:04,654 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b355a132009f4d199985a25eb4b6abf1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b355a132009f4d199985a25eb4b6abf1 2023-04-18 18:55:04,655 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/2a14f50354f741c1b59bfb04a8e02558 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/2a14f50354f741c1b59bfb04a8e02558 2023-04-18 18:55:04,656 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/c8f83b60c70b4a7dbf22ad6768851bcc to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/c8f83b60c70b4a7dbf22ad6768851bcc 2023-04-18 18:55:04,657 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/c801ebe22f6849648b750b70c0d3b068 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/c801ebe22f6849648b750b70c0d3b068 2023-04-18 18:55:04,658 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/041e601490a7465ba23943a14250da5c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/041e601490a7465ba23943a14250da5c 2023-04-18 18:55:04,660 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b0334c76c3c5439499b61b051d44dae3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b0334c76c3c5439499b61b051d44dae3 2023-04-18 18:55:04,661 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/afc8eb42667c410797456e9e1c56a05d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/afc8eb42667c410797456e9e1c56a05d 2023-04-18 18:55:04,662 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/4a3781aa449a4bcaa1fc27c6db83f0db to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/4a3781aa449a4bcaa1fc27c6db83f0db 2023-04-18 18:55:04,663 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/a9aa45f6aa0947528c1363c2ef98545e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/a9aa45f6aa0947528c1363c2ef98545e 2023-04-18 18:55:04,664 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/e73c2c866c6040bbbb6d1204224ef1d6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/e73c2c866c6040bbbb6d1204224ef1d6 2023-04-18 18:55:04,666 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/73d0655bd8f949699d4e6eb0030b3434 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/73d0655bd8f949699d4e6eb0030b3434 2023-04-18 18:55:04,667 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/a1fba916d7cd4e1eaca275147f7133dc to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/a1fba916d7cd4e1eaca275147f7133dc 2023-04-18 18:55:04,668 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/5f5851c62aa64aec8bd1ce9400719acc to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/5f5851c62aa64aec8bd1ce9400719acc 2023-04-18 18:55:04,669 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/be08611e9fb64373a733fdd0e40b1f51 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/be08611e9fb64373a733fdd0e40b1f51 2023-04-18 18:55:04,670 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/8d76954c6e60475cb94145334cf2cb0c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/8d76954c6e60475cb94145334cf2cb0c 2023-04-18 18:55:04,671 DEBUG [StoreCloser-TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/3c7a7d9689d74f878ac3d2fa27889eb5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/3c7a7d9689d74f878ac3d2fa27889eb5 2023-04-18 18:55:04,675 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/recovered.edits/1677.seqid, newMaxSeqId=1677, maxSeqId=1 2023-04-18 18:55:04,677 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a. 2023-04-18 18:55:04,677 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e3c3c3367729f1159cdc97339f70376a: 2023-04-18 18:55:04,678 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:55:04,678 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=e3c3c3367729f1159cdc97339f70376a, regionState=CLOSED 2023-04-18 18:55:04,679 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844104678"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681844104678"}]},"ts":"1681844104678"} 2023-04-18 18:55:04,680 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-18 18:55:04,681 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure e3c3c3367729f1159cdc97339f70376a, server=jenkins-hbase4.apache.org,34053,1681844018952 in 2.5560 sec 2023-04-18 18:55:04,682 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-18 18:55:04,682 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e3c3c3367729f1159cdc97339f70376a, UNASSIGN in 2.5600 sec 2023-04-18 18:55:04,682 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844104682"}]},"ts":"1681844104682"} 2023-04-18 18:55:04,683 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-18 18:55:04,685 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-18 18:55:04,686 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.5710 sec 2023-04-18 18:55:06,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-18 18:55:06,223 INFO [Listener at localhost/36951] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-18 18:55:06,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-18 18:55:06,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:55:06,226 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:55:06,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-18 18:55:06,227 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:55:06,231 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:55:06,233 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A, FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B, FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C, FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/recovered.edits] 2023-04-18 18:55:06,236 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/496e09bb198e460389981c2d9c54ff48 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/496e09bb198e460389981c2d9c54ff48 2023-04-18 18:55:06,237 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/c031218c82af486bb69525129fab1ade to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/c031218c82af486bb69525129fab1ade 2023-04-18 18:55:06,238 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/c3e9516de80a4f129f76436b8dd398ad to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/c3e9516de80a4f129f76436b8dd398ad 2023-04-18 18:55:06,239 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/f2ba448e302a40618dd8b5e048ecc125 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/A/f2ba448e302a40618dd8b5e048ecc125 2023-04-18 18:55:06,241 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/2c3565896f3646eda74b13dbc671ce5a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/2c3565896f3646eda74b13dbc671ce5a 2023-04-18 18:55:06,242 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/49d88fa6315f4f339f560819698485d1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/49d88fa6315f4f339f560819698485d1 2023-04-18 18:55:06,243 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/d2663579d9d74c2bbf2fd96577048310 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/d2663579d9d74c2bbf2fd96577048310 2023-04-18 18:55:06,244 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/d345e079bfad4d7da8090ccdd6d3d738 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/B/d345e079bfad4d7da8090ccdd6d3d738 2023-04-18 18:55:06,246 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/45fb9b1d064546ca87451b5a5e673fba to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/45fb9b1d064546ca87451b5a5e673fba 2023-04-18 18:55:06,247 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/a6c035d4c2974f5c93d759524b64f067 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/a6c035d4c2974f5c93d759524b64f067 2023-04-18 18:55:06,248 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b309b8970e7142149301061cd022f691 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/b309b8970e7142149301061cd022f691 2023-04-18 18:55:06,249 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/ba490ea8d1644eaaaadd112293526816 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/C/ba490ea8d1644eaaaadd112293526816 2023-04-18 18:55:06,251 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/recovered.edits/1677.seqid to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a/recovered.edits/1677.seqid 2023-04-18 18:55:06,252 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/e3c3c3367729f1159cdc97339f70376a 2023-04-18 18:55:06,252 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 18:55:06,254 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:55:06,255 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-18 18:55:06,256 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-18 18:55:06,257 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:55:06,257 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-18 18:55:06,257 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681844106257"}]},"ts":"9223372036854775807"} 2023-04-18 18:55:06,258 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-18 18:55:06,258 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => e3c3c3367729f1159cdc97339f70376a, NAME => 'TestAcidGuarantees,,1681844079783.e3c3c3367729f1159cdc97339f70376a.', STARTKEY => '', ENDKEY => ''}] 2023-04-18 18:55:06,258 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-18 18:55:06,259 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681844106258"}]},"ts":"9223372036854775807"} 2023-04-18 18:55:06,260 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-18 18:55:06,261 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:55:06,262 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 37 msec 2023-04-18 18:55:06,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-18 18:55:06,328 INFO [Listener at localhost/36951] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-18 18:55:06,339 INFO [Listener at localhost/36951] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227 (was 232), OpenFileDescriptor=565 (was 553) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=375 (was 378), ProcessCount=169 (was 169), AvailableMemoryMB=6587 (was 6615) 2023-04-18 18:55:06,350 INFO [Listener at localhost/36951] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227, OpenFileDescriptor=565, MaxFileDescriptor=60000, SystemLoadAverage=375, ProcessCount=169, AvailableMemoryMB=6586 2023-04-18 18:55:06,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-18 18:55:06,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-18 18:55:06,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-18 18:55:06,354 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-18 18:55:06,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-18 18:55:06,355 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-18 18:55:06,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-18 18:55:06,356 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:06,357 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa empty. 2023-04-18 18:55:06,357 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:06,357 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 18:55:06,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-18 18:55:06,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-18 18:55:06,764 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-18 18:55:06,766 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => a9efbb28428802c9e45d9365ba7ee3aa, NAME => 'TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp 2023-04-18 18:55:06,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-18 18:55:07,173 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 18:55:07,173 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing a9efbb28428802c9e45d9365ba7ee3aa, disabling compactions & flushes 2023-04-18 18:55:07,173 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:07,173 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:07,173 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. after waiting 0 ms 2023-04-18 18:55:07,173 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:07,173 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:07,173 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:07,176 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-18 18:55:07,176 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844107176"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681844107176"}]},"ts":"1681844107176"} 2023-04-18 18:55:07,177 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-18 18:55:07,178 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-18 18:55:07,178 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844107178"}]},"ts":"1681844107178"} 2023-04-18 18:55:07,178 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-18 18:55:07,181 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9efbb28428802c9e45d9365ba7ee3aa, ASSIGN}] 2023-04-18 18:55:07,182 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9efbb28428802c9e45d9365ba7ee3aa, ASSIGN 2023-04-18 18:55:07,183 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9efbb28428802c9e45d9365ba7ee3aa, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34053,1681844018952; forceNewPlan=false, retain=false 2023-04-18 18:55:07,334 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=a9efbb28428802c9e45d9365ba7ee3aa, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:07,334 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844107334"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681844107334"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681844107334"}]},"ts":"1681844107334"} 2023-04-18 18:55:07,335 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952}] 2023-04-18 18:55:07,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-18 18:55:07,490 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:07,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a9efbb28428802c9e45d9365ba7ee3aa, NAME => 'TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa.', STARTKEY => '', ENDKEY => ''} 2023-04-18 18:55:07,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:07,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 18:55:07,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:07,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:07,491 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:07,493 DEBUG [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A 2023-04-18 18:55:07,493 DEBUG [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A 2023-04-18 18:55:07,493 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:55:07,493 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a9efbb28428802c9e45d9365ba7ee3aa columnFamilyName A 2023-04-18 18:55:07,494 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] regionserver.HStore(310): Store=a9efbb28428802c9e45d9365ba7ee3aa/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:55:07,494 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:07,495 DEBUG [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B 2023-04-18 18:55:07,495 DEBUG [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B 2023-04-18 18:55:07,495 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:55:07,495 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a9efbb28428802c9e45d9365ba7ee3aa columnFamilyName B 2023-04-18 18:55:07,496 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] regionserver.HStore(310): Store=a9efbb28428802c9e45d9365ba7ee3aa/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:55:07,496 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:07,497 DEBUG [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C 2023-04-18 18:55:07,497 DEBUG [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C 2023-04-18 18:55:07,497 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:55:07,497 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a9efbb28428802c9e45d9365ba7ee3aa columnFamilyName C 2023-04-18 18:55:07,497 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] regionserver.HStore(310): Store=a9efbb28428802c9e45d9365ba7ee3aa/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:55:07,497 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:07,498 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:07,498 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:07,500 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 18:55:07,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:07,502 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-18 18:55:07,502 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a9efbb28428802c9e45d9365ba7ee3aa; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71368596, jitterRate=0.06347495317459106}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 18:55:07,502 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:07,503 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., pid=37, masterSystemTime=1681844107487 2023-04-18 18:55:07,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:07,504 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:07,505 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=a9efbb28428802c9e45d9365ba7ee3aa, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:07,505 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844107504"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681844107504"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681844107504"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681844107504"}]},"ts":"1681844107504"} 2023-04-18 18:55:07,507 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-18 18:55:07,507 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 in 171 msec 2023-04-18 18:55:07,508 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-18 18:55:07,508 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9efbb28428802c9e45d9365ba7ee3aa, ASSIGN in 326 msec 2023-04-18 18:55:07,509 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-18 18:55:07,509 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844107509"}]},"ts":"1681844107509"} 2023-04-18 18:55:07,509 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-18 18:55:07,512 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-18 18:55:07,513 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1590 sec 2023-04-18 18:55:08,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-18 18:55:08,460 INFO [Listener at localhost/36951] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-18 18:55:08,460 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x118e131f to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:08,465 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1cee66b5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:08,466 DEBUG [Listener at localhost/36951] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:08,468 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47214, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:08,469 DEBUG [Listener at localhost/36951] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-18 18:55:08,470 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44322, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-18 18:55:08,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-18 18:55:08,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-18 18:55:08,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-18 18:55:08,881 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-18 18:55:08,882 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-18 18:55:08,882 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-18 18:55:08,884 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-18 18:55:08,887 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9efbb28428802c9e45d9365ba7ee3aa, REOPEN/MOVE}] 2023-04-18 18:55:08,887 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9efbb28428802c9e45d9365ba7ee3aa, REOPEN/MOVE 2023-04-18 18:55:08,888 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a9efbb28428802c9e45d9365ba7ee3aa, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:08,888 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844108888"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681844108888"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681844108888"}]},"ts":"1681844108888"} 2023-04-18 18:55:08,889 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952}] 2023-04-18 18:55:09,040 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:09,041 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a9efbb28428802c9e45d9365ba7ee3aa, disabling compactions & flushes 2023-04-18 18:55:09,041 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:09,041 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:09,041 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. after waiting 0 ms 2023-04-18 18:55:09,041 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:09,045 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-18 18:55:09,045 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:09,045 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:09,045 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: a9efbb28428802c9e45d9365ba7ee3aa to self. 2023-04-18 18:55:09,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:09,047 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a9efbb28428802c9e45d9365ba7ee3aa, regionState=CLOSED 2023-04-18 18:55:09,047 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844109047"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681844109047"}]},"ts":"1681844109047"} 2023-04-18 18:55:09,049 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-18 18:55:09,049 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 in 159 msec 2023-04-18 18:55:09,050 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9efbb28428802c9e45d9365ba7ee3aa, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,34053,1681844018952; forceNewPlan=false, retain=true 2023-04-18 18:55:09,200 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a9efbb28428802c9e45d9365ba7ee3aa, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,200 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844109200"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681844109200"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681844109200"}]},"ts":"1681844109200"} 2023-04-18 18:55:09,201 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952}] 2023-04-18 18:55:09,355 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:09,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a9efbb28428802c9e45d9365ba7ee3aa, NAME => 'TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa.', STARTKEY => '', ENDKEY => ''} 2023-04-18 18:55:09,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:09,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 18:55:09,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:09,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:09,358 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:09,358 DEBUG [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A 2023-04-18 18:55:09,359 DEBUG [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A 2023-04-18 18:55:09,359 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:55:09,359 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a9efbb28428802c9e45d9365ba7ee3aa columnFamilyName A 2023-04-18 18:55:09,360 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] regionserver.HStore(310): Store=a9efbb28428802c9e45d9365ba7ee3aa/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:55:09,361 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:09,361 DEBUG [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B 2023-04-18 18:55:09,361 DEBUG [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B 2023-04-18 18:55:09,361 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:55:09,362 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a9efbb28428802c9e45d9365ba7ee3aa columnFamilyName B 2023-04-18 18:55:09,362 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] regionserver.HStore(310): Store=a9efbb28428802c9e45d9365ba7ee3aa/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:55:09,362 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:09,363 DEBUG [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C 2023-04-18 18:55:09,363 DEBUG [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C 2023-04-18 18:55:09,363 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:55:09,363 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a9efbb28428802c9e45d9365ba7ee3aa columnFamilyName C 2023-04-18 18:55:09,363 INFO [StoreOpener-a9efbb28428802c9e45d9365ba7ee3aa-1] regionserver.HStore(310): Store=a9efbb28428802c9e45d9365ba7ee3aa/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:55:09,363 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:09,364 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:09,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:09,367 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 18:55:09,368 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:09,368 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a9efbb28428802c9e45d9365ba7ee3aa; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59983872, jitterRate=-0.106170654296875}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 18:55:09,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:09,369 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., pid=42, masterSystemTime=1681844109353 2023-04-18 18:55:09,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:09,370 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:09,370 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a9efbb28428802c9e45d9365ba7ee3aa, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,371 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844109370"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681844109370"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681844109370"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681844109370"}]},"ts":"1681844109370"} 2023-04-18 18:55:09,372 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-18 18:55:09,373 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 in 170 msec 2023-04-18 18:55:09,374 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-18 18:55:09,374 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9efbb28428802c9e45d9365ba7ee3aa, REOPEN/MOVE in 485 msec 2023-04-18 18:55:09,375 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-18 18:55:09,375 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 490 msec 2023-04-18 18:55:09,376 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 903 msec 2023-04-18 18:55:09,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-18 18:55:09,378 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x2deaf1fa to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:09,382 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4c1d19b3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:09,382 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x6e7e18ef to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:09,386 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14e151fb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:09,387 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x5b7d4a51 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:09,393 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@455a88b1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:09,394 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x4335c1f2 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:09,397 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@722b2407, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:09,398 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x6bbc9fc8 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:09,402 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66bf91c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:09,402 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x6846dddb to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:09,406 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@313c7ad6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:09,406 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x7ea43962 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:09,411 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20d28b68, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:09,411 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x2abd7961 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:09,416 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@750d2a7d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:09,416 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x660630e8 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:09,419 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@27313b38, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:09,419 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x793239ee to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:09,423 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ee4c750, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:09,425 DEBUG [hconnection-0xc1c9104-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:09,426 DEBUG [hconnection-0x662551c6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:09,426 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47218, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:09,430 DEBUG [hconnection-0x52caffbe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:09,431 DEBUG [hconnection-0x4c3cf06d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:09,431 DEBUG [hconnection-0x252eff3e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:09,431 DEBUG [hconnection-0xf403764-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:09,431 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47228, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:09,432 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47242, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:09,437 DEBUG [hconnection-0x4873e94f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:09,437 DEBUG [hconnection-0xa8ad4bd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:09,440 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47236, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:09,440 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47250, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:09,440 DEBUG [hconnection-0x4c07871f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:09,440 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47246, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:09,440 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47258, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:09,440 DEBUG [hconnection-0x19062ef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:09,442 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47266, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:09,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:09,444 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47276, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:09,446 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47270, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:09,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:09,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:09,450 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:09,450 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:09,450 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:09,450 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:09,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:09,452 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-18 18:55:09,453 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,454 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,455 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:09,455 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:09,455 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:09,455 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:09,455 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:09,455 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:09,457 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:09,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:09,457 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,457 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,457 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:09,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:09,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:09,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:09,461 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:09,461 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:09,461 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:09,461 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:09,461 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:09,461 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:09,461 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:09,461 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:09,461 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:09,461 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:09,461 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:09,461 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,462 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:09,462 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:09,463 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,463 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:09,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:09,464 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,464 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:09,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:09,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:09,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:09,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:09,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e9b253f[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:09,465 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:09,465 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,466 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,467 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:09,467 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:09,467 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:09,467 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,467 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:09,468 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,468 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:09,468 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:09,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:09,468 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:09,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:09,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:09,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:09,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:09,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:09,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47242 deadline: 1681844169470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:09,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844169470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:09,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47228 deadline: 1681844169470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:09,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:09,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844169471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:09,478 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:09,478 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:09,478 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:09,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:09,478 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:09,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47258 deadline: 1681844169472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,478 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:09,480 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,480 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,481 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:09,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:09,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:09,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,481 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:09,485 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:09,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,487 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,487 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,487 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:09,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:09,487 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:09,487 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:09,487 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:09,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:09,487 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:09,487 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:09,487 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,487 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:09,488 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,488 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:09,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:09,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:09,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:09,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47242 deadline: 1681844169578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:09,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844169578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:09,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47228 deadline: 1681844169578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:09,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47258 deadline: 1681844169579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:09,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844169579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,588 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:09,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:09,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:09,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:09,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:09,592 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:09,592 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:09,592 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:09,592 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:09,593 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,593 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,594 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:09,594 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,596 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,596 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:09,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:09,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,597 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:09,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:09,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:09,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:09,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:09,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:09,604 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:09,604 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:09,604 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:09,604 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:09,604 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:09,604 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:09,604 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:09,604 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:09,604 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,610 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:09,611 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:09,612 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,612 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:09,613 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,613 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:09,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:09,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7043136e[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:09,614 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:09,614 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,616 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,616 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:09,616 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:09,616 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:09,616 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,616 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:09,617 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:09,617 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:09,617 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,618 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:09,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:09,620 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:09,620 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:09,620 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:09,620 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:09,620 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:09,622 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,622 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,622 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:09,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:09,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:09,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:09,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:09,623 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:09,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:09,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,624 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:09,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,628 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,628 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,628 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,628 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:09,628 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:09,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:09,628 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:09,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:09,628 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:09,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:09,628 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:09,628 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,628 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,628 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:09,628 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:09,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:09,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:09,728 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:09,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:09,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:09,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:09,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:09,732 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:09,732 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:09,732 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:09,732 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:09,732 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,732 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,734 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:09,734 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,734 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,734 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:09,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:09,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:09,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:09,735 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:09,735 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:09,735 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:09,736 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:09,736 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:09,736 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:09,736 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:09,736 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:09,736 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:09,736 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:09,736 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:09,736 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,736 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:09,736 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:09,737 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,737 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:09,738 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,738 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:09,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:09,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@407dcefd[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:09,739 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:09,739 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,740 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,740 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:09,740 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:09,740 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,740 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:09,740 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:09,742 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:09,742 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:09,742 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:09,742 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:09,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:09,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:09,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:09,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:09,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:09,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:09,744 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:09,745 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:09,745 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:09,745 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:09,745 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:09,748 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:09,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:09,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:09,749 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:09,750 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,750 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:09,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:09,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:09,750 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:09,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,756 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,756 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:09,756 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:09,756 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,756 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,756 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:09,756 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:09,756 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:09,756 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:09,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:09,756 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,756 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:09,756 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:09,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:09,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:09,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47242 deadline: 1681844169781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:09,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47258 deadline: 1681844169781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:09,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844169782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:09,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844169782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:09,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47228 deadline: 1681844169782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,857 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:09,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:09,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:09,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:09,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:09,860 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:09,860 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:09,860 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:09,860 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:09,861 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,861 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,862 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,862 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:09,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:09,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:09,862 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,862 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:09,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:09,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:09,863 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:09,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:09,863 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:09,864 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:09,864 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:09,864 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:09,864 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:09,864 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:09,864 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:09,864 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:09,864 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:09,864 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:09,864 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:09,864 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,864 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:09,864 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:09,866 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,866 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:09,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:09,866 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,866 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:09,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:09,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:09,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ef29336[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:09,868 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:09,868 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,870 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,870 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:09,870 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:09,870 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:09,870 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,871 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:09,873 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,873 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:09,873 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:09,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:09,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:09,873 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:09,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:09,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:09,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:09,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:09,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:09,876 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:09,876 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:09,876 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:09,877 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:09,877 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:09,878 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418ad282f41950b4879b4fb0a4c50870ba6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418ad282f41950b4879b4fb0a4c50870ba6 2023-04-18 18:55:09,879 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,879 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,879 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:09,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:09,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:09,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:09,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:09,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:09,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,881 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:09,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,882 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:09,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:09,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,886 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,886 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:09,886 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:09,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:09,886 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,886 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,886 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,886 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:09,886 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:09,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:09,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:09,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:09,886 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:09,886 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,886 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:09,887 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:09,987 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:09,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:09,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:09,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:09,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:09,990 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:09,990 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:09,991 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:09,991 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:09,991 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,991 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,994 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,994 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:09,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:09,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:09,995 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,995 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:09,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:09,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:09,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:09,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:09,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:09,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:09,996 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:09,996 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:09,996 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:09,996 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:09,996 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:09,996 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:09,996 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:09,996 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:09,996 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:09,996 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:09,996 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:09,996 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,996 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:09,997 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:09,999 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,999 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:09,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:09,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:09,999 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:09,999 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:10,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:10,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69a7fd0a[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:10,002 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:10,002 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,003 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,004 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:10,004 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,004 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:10,004 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:10,004 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:10,005 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,005 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:10,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,005 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:10,006 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:10,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:10,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:10,008 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:10,008 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:10,008 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:10,008 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:10,008 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:10,010 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,010 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:10,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,010 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:10,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:10,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,011 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,014 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,016 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,016 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:10,016 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:10,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:10,016 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,016 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,016 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,016 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:10,016 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:10,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:10,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:10,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:10,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:10,016 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:10,016 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,017 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:10,017 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:10,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:10,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47242 deadline: 1681844170086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:10,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844170087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:10,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47228 deadline: 1681844170087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:10,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47258 deadline: 1681844170087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:10,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844170087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,117 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:10,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:10,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:10,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:10,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:10,121 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:10,121 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:10,121 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:10,121 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:10,121 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,121 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,123 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:10,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:10,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:10,123 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,123 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,123 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:10,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,124 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:10,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:10,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:10,124 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:10,124 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:10,124 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:10,124 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:10,125 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:10,125 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:10,125 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:10,125 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:10,125 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:10,125 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:10,125 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:10,125 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,125 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:10,125 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:10,126 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,126 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,127 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,127 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:10,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:10,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1175b33[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:10,128 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:10,128 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,129 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,129 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:10,130 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:10,130 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:10,130 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,130 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:10,132 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,132 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:10,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,132 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:10,133 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:10,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:10,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:10,135 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:10,135 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:10,135 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:10,135 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:10,135 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:10,137 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,137 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,137 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:10,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,137 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:10,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:10,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,138 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,141 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,141 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,141 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:10,141 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:10,141 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,141 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:10,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:10,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:10,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:10,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:10,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:10,141 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,141 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,141 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,141 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,142 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:10,142 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:10,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:10,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:10,203 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-18 18:55:10,242 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:10,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:10,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:10,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:10,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:10,246 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:10,246 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:10,246 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:10,246 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:10,246 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,246 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,251 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,251 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:10,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:10,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:10,251 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,251 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:10,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:10,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:10,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:10,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:10,253 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:10,253 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:10,253 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:10,254 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:10,254 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:10,254 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:10,254 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:10,254 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:10,254 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:10,254 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:10,254 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:10,254 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,254 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:10,254 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:10,256 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,257 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,257 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,257 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:10,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:10,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f3367ca[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:10,258 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:10,258 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,259 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,259 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:10,259 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:10,259 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:10,259 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,259 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:10,261 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,261 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:10,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,261 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:10,261 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:10,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:10,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:10,263 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:10,263 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:10,263 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:10,263 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:10,264 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:10,265 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,265 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,265 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:10,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:10,265 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:10,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,266 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,268 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,268 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:10,268 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,269 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,269 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,269 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,269 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,269 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:10,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:10,269 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:10,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:10,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:10,269 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:10,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:10,269 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:10,269 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:10,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:10,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:10,284 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/f074f894c6a643d0841190813863f2ae 2023-04-18 18:55:10,369 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:10,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:10,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:10,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:10,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:10,373 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:10,373 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:10,373 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:10,373 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:10,373 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,373 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,375 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:10,375 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:10,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:10,376 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,376 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:10,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:10,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:10,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:10,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:10,377 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:10,377 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:10,377 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:10,377 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:10,377 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:10,377 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:10,377 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:10,377 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:10,377 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:10,377 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:10,377 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:10,377 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,377 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:10,377 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:10,380 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,380 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,380 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:10,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:10,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35c23a39[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:10,382 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:10,382 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,383 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,383 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:10,383 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:10,383 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:10,383 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,384 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:10,385 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,385 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:10,385 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:10,385 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:10,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:10,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:10,387 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:10,387 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:10,387 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:10,387 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:10,387 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:10,389 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,389 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,389 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:10,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:10,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:10,390 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,391 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,394 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,394 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,394 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,394 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,394 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,394 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,394 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:10,395 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:10,394 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:10,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:10,395 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:10,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:10,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:10,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:10,395 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:10,395 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:10,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:10,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:10,495 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:10,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:10,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:10,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:10,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:10,498 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:10,498 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:10,498 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:10,498 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:10,499 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,499 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,500 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:10,500 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:10,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:10,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,501 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,501 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:10,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:10,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:10,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:10,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:10,502 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:10,502 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:10,502 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:10,502 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:10,502 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:10,502 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:10,502 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:10,502 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:10,502 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:10,502 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:10,502 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:10,502 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,502 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:10,502 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:10,504 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,504 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,505 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,505 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:10,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:10,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@635bde26[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:10,506 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:10,506 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,507 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,508 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:10,508 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:10,508 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:10,508 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,508 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:10,510 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:10,510 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:10,510 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,510 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:10,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:10,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:10,512 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:10,512 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:10,512 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:10,512 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:10,512 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:10,514 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,514 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,514 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:10,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,514 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:10,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:10,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,515 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,518 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,518 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,518 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:10,518 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:10,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:10,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:10,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:10,518 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,518 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:10,518 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,518 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:10,518 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:10,518 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:10,518 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:10,518 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:10,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:10,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:10,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47228 deadline: 1681844170591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:10,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844170591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:10,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:10,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47242 deadline: 1681844170592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47258 deadline: 1681844170593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:10,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844170593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,618 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:10,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:10,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:10,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:10,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:10,623 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:10,623 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:10,623 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:10,623 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:10,623 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,623 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,624 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:10,624 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:10,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:10,625 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,625 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:10,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:10,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:10,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:10,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:10,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:10,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:10,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:10,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:10,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:10,626 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:10,626 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:10,626 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:10,626 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:10,627 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:10,627 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:10,627 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,627 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:10,627 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:10,629 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,629 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,629 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,629 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:10,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:10,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58aecfbd[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:10,630 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:10,630 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,635 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,635 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,635 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:10,635 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:10,635 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:10,635 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:10,636 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:10,637 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,637 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:10,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,637 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:10,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:10,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:10,639 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:10,639 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:10,639 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:10,639 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:10,639 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:10,641 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,641 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:10,641 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:10,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:10,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,642 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,643 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,647 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,647 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,647 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,647 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:10,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:10,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:10,647 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:10,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:10,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:10,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:10,647 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:10,647 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:10,647 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:10,647 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,647 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,647 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,647 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:10,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:10,695 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/1a05d716fba546d6b402df3a9f5d3a22 2023-04-18 18:55:10,747 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:10,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:10,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:10,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:10,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:10,753 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:10,753 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:10,753 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:10,753 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:10,753 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,753 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,755 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:10,755 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:10,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:10,755 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,755 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:10,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:10,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:10,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:10,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:10,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:10,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:10,757 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:10,768 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:10,768 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:10,768 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:10,768 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:10,768 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:10,768 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:10,768 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:10,768 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:10,768 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,768 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:10,768 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:10,770 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,770 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,771 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,771 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:10,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:10,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c97ee55[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:10,773 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:10,773 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,775 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,775 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:10,775 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:10,775 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:10,775 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,775 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:10,778 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:10,778 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:10,778 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:10,778 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:10,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:10,783 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:10,783 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:10,783 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:10,783 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:10,783 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:10,785 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,785 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:10,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,785 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:10,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:10,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,786 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,787 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,790 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:10,790 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,790 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,790 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,790 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:10,790 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:10,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:10,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:10,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:10,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:10,791 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:10,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:10,790 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,791 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,791 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,791 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,791 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:10,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:10,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:10,891 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:10,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:10,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:10,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:10,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:10,895 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:10,896 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:10,896 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:10,896 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:10,896 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,896 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,897 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:10,897 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:10,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:10,898 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,898 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:10,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:10,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,899 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:10,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:10,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:10,899 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:10,899 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:10,899 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:10,899 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:10,899 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:10,900 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:10,900 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:10,900 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:10,900 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:10,900 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:10,900 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:10,900 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,900 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:10,900 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:10,902 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,902 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,903 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,903 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:10,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:10,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48a0120c[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:10,904 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:10,904 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,905 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,906 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:10,906 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:10,906 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:10,906 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,906 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:10,908 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:10,908 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:10,908 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,908 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:10,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:10,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:10,912 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:10,912 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:10,912 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:10,912 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:10,912 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:10,915 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,915 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,915 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:10,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:10,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:10,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,915 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:10,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:10,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:10,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:10,916 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:10,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:10,919 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,919 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,919 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:10,919 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:10,919 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:10,920 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:10,920 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:10,919 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,920 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:10,920 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:10,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:10,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:10,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:10,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:10,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:10,921 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:10,921 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:10,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:10,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:11,021 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:11,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:11,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:11,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:11,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:11,026 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:11,026 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:11,026 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:11,026 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:11,026 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,026 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:11,028 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:11,028 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:11,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:11,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:11,028 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,028 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:11,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:11,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:11,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:11,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:11,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:11,029 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:11,029 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:11,029 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:11,029 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:11,030 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:11,030 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:11,030 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:11,030 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:11,030 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:11,030 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:11,030 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:11,030 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,030 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:11,030 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:11,032 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:11,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:11,032 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:11,033 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,033 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:11,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:11,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:11,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:11,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:11,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75374fd0[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:11,034 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:11,034 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,035 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,036 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:11,036 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:11,036 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:11,036 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,036 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:11,041 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:11,041 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:11,041 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:11,041 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:11,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:11,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:11,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:11,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:11,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:11,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:11,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:11,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:11,044 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:11,045 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:11,045 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:11,045 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:11,045 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:11,046 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,046 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:11,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:11,046 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:11,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,047 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:11,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:11,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:11,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:11,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:11,048 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:11,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:11,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:11,050 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,050 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:11,050 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:11,050 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:11,050 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,051 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:11,051 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:11,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:11,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:11,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:11,051 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:11,050 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:11,051 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:11,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:11,051 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,051 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:11,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:11,110 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/ccd6d5bd04a341f683195239a4de2a5a 2023-04-18 18:55:11,116 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/f074f894c6a643d0841190813863f2ae as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/f074f894c6a643d0841190813863f2ae 2023-04-18 18:55:11,120 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/f074f894c6a643d0841190813863f2ae, entries=150, sequenceid=16, filesize=25.1 K 2023-04-18 18:55:11,121 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/1a05d716fba546d6b402df3a9f5d3a22 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/1a05d716fba546d6b402df3a9f5d3a22 2023-04-18 18:55:11,127 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/1a05d716fba546d6b402df3a9f5d3a22, entries=150, sequenceid=16, filesize=11.6 K 2023-04-18 18:55:11,128 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/ccd6d5bd04a341f683195239a4de2a5a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/ccd6d5bd04a341f683195239a4de2a5a 2023-04-18 18:55:11,133 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/ccd6d5bd04a341f683195239a4de2a5a, entries=150, sequenceid=16, filesize=11.6 K 2023-04-18 18:55:11,136 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a9efbb28428802c9e45d9365ba7ee3aa in 1684ms, sequenceid=16, compaction requested=false 2023-04-18 18:55:11,137 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:11,151 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:11,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:11,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:11,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:11,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:11,156 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:11,156 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:11,156 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:11,156 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:11,157 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,157 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:11,160 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:11,160 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:11,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:11,160 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,160 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:11,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:11,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:11,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:11,161 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:11,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:11,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:11,167 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:11,167 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:11,167 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:11,172 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:11,172 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:11,172 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 18:55:11,173 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:11,173 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:11,173 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:11,173 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:11,173 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:11,173 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:11,212 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418df9f2ec7d24e4fcaabdba7ee99191afb to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418df9f2ec7d24e4fcaabdba7ee99191afb 2023-04-18 18:55:11,224 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/199c75b353ba4aeaa5100720ca0ae64d 2023-04-18 18:55:11,257 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/af8260e1210741789f2cb61c4b00febe 2023-04-18 18:55:11,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:11,599 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:11,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:11,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47242 deadline: 1681844171608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:11,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47258 deadline: 1681844171610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:11,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47228 deadline: 1681844171611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:11,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844171612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:11,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844171613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,690 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/80cd6fe4f0c344ad8fef4fe7dc54399b 2023-04-18 18:55:11,707 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/199c75b353ba4aeaa5100720ca0ae64d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/199c75b353ba4aeaa5100720ca0ae64d 2023-04-18 18:55:11,711 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/199c75b353ba4aeaa5100720ca0ae64d, entries=150, sequenceid=36, filesize=25.1 K 2023-04-18 18:55:11,713 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/af8260e1210741789f2cb61c4b00febe as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/af8260e1210741789f2cb61c4b00febe 2023-04-18 18:55:11,718 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/af8260e1210741789f2cb61c4b00febe, entries=150, sequenceid=36, filesize=11.6 K 2023-04-18 18:55:11,720 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/80cd6fe4f0c344ad8fef4fe7dc54399b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/80cd6fe4f0c344ad8fef4fe7dc54399b 2023-04-18 18:55:11,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:11,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47228 deadline: 1681844171717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:11,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47258 deadline: 1681844171717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,724 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/80cd6fe4f0c344ad8fef4fe7dc54399b, entries=150, sequenceid=36, filesize=11.6 K 2023-04-18 18:55:11,725 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for a9efbb28428802c9e45d9365ba7ee3aa in 553ms, sequenceid=36, compaction requested=false 2023-04-18 18:55:11,725 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:11,725 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:11,725 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:11,725 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:11,725 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,728 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:11,728 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:11,729 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,729 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:11,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:11,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:11,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:11,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:11,731 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,731 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:11,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:11,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:11,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b4aa5ec[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:11,732 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:11,732 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:11,732 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-18 18:55:11,732 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:11,732 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:11,733 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:11,733 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:11,733 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:11,733 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:11,733 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,734 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:11,734 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:11,734 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:11,734 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,734 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:11,736 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,736 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:11,736 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:11,737 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:11,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:11,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:11,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:11,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:11,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:11,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:11,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:11,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:11,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:11,739 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:11,739 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:11,739 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:11,739 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:11,740 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:11,748 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,748 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:11,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,748 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:11,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:11,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:11,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,749 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:11,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:11,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:11,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,751 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:11,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:11,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,753 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:11,753 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:11,753 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,753 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:11,753 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:11,753 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,753 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:11,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:11,753 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:11,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:11,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:11,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:11,753 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,754 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:11,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:11,754 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:11,754 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:11,781 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304188837bcb52a994edb919144f20cfec895 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304188837bcb52a994edb919144f20cfec895 2023-04-18 18:55:11,790 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/7b7bac1a14dc43c993bcb5a6a77eb633 2023-04-18 18:55:11,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:11,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844171786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:11,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844171792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,854 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:11,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:11,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:11,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:11,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:11,858 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:11,858 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:11,858 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:11,858 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:11,858 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,858 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:11,860 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,860 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:11,860 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,860 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:11,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:11,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:11,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:11,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:11,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:11,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:11,863 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:11,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:11,863 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:11,863 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:11,863 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:11,863 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:11,863 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:11,863 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:11,863 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:11,863 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:11,863 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:11,863 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:11,863 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:11,864 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,864 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:11,864 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:11,865 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,865 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:11,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:11,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:11,867 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,867 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:11,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:11,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:11,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:11,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:11,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8fc0d96[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:11,870 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:11,870 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,872 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,873 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:11,873 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:11,873 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:11,873 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,873 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:11,876 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,876 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:11,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:11,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:11,876 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:11,876 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:11,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:11,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:11,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:11,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:11,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:11,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:11,884 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:11,884 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:11,884 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:11,884 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:11,884 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:11,887 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,887 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:11,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:11,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:11,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,888 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:11,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:11,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:11,891 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:11,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:11,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:11,893 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:11,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:11,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,900 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:11,900 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:11,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:11,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:11,900 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,900 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:11,900 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:11,900 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,900 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,900 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:11,900 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:11,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:11,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844171895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,901 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:11,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:11,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:11,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:11,901 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:11,901 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:11,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:11,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:11,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:11,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844171901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:11,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47228 deadline: 1681844171925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:11,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:11,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47258 deadline: 1681844171925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,001 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:12,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:12,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:12,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:12,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:12,005 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:12,005 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:12,005 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:12,005 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:12,007 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:12,007 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:12,010 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:12,010 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:12,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:12,010 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,010 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:12,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:12,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:12,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:12,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:12,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:12,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:12,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:12,011 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:12,011 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:12,011 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:12,012 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:12,012 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:12,012 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:12,012 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:12,012 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:12,012 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:12,012 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:12,012 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:12,012 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,012 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:12,012 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:12,014 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:12,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:12,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,015 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:12,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:12,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:12,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40044ef7[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:12,016 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:12,017 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:12,017 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:12,017 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:12,018 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:12,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:12,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:12,018 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:12,018 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:12,018 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:12,018 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,018 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:12,019 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,019 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:12,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:12,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:12,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:12,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:12,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:12,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:12,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:12,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:12,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:12,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:12,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:12,022 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:12,022 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:12,022 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:12,022 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:12,022 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:12,025 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:12,025 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:12,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:12,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:12,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:12,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:12,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:12,025 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:12,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:12,026 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:12,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:12,027 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:12,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:12,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,036 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:12,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:12,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:12,037 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:12,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:12,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:12,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:12,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:12,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:12,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:12,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:12,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:12,037 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:12,037 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:12,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:12,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:12,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:12,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:12,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844172103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:12,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844172105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,137 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:12,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:12,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:12,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:12,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:12,141 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:12,141 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:12,141 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:12,141 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:12,142 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:12,142 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:12,143 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,143 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:12,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:12,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:12,143 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,143 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:12,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:12,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:12,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:12,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:12,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:12,145 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:12,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:12,145 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:12,145 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:12,145 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:12,145 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:12,147 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,147 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:12,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:12,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:12,147 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:12,147 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:12,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:12,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:12,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:12,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:12,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:12,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f02d0e8[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:12,150 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:12,150 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:12,152 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:12,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:12,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:12,152 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:12,152 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:12,152 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:12,152 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,153 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:12,154 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:12,154 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:12,154 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,154 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:12,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:12,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:12,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:12,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:12,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:12,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:12,157 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:12,157 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:12,157 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:12,157 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:12,157 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:12,158 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:12,158 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:12,158 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:12,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:12,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:12,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:12,159 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:12,160 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,163 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,163 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:12,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:12,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:12,163 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:12,163 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,163 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:12,163 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:12,163 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:12,164 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:12,164 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:12,163 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:12,164 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:12,163 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:12,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:12,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:12,214 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/c9ff752f5eb34ea78250f286b103a5e4 2023-04-18 18:55:12,226 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/163c95d502b74223865c5446ca2ad0bd 2023-04-18 18:55:12,232 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/7b7bac1a14dc43c993bcb5a6a77eb633 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/7b7bac1a14dc43c993bcb5a6a77eb633 2023-04-18 18:55:12,234 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:12,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47228 deadline: 1681844172231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,234 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:12,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47258 deadline: 1681844172231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,236 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/7b7bac1a14dc43c993bcb5a6a77eb633, entries=200, sequenceid=50, filesize=31.9 K 2023-04-18 18:55:12,237 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/c9ff752f5eb34ea78250f286b103a5e4 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/c9ff752f5eb34ea78250f286b103a5e4 2023-04-18 18:55:12,242 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/c9ff752f5eb34ea78250f286b103a5e4, entries=150, sequenceid=50, filesize=11.6 K 2023-04-18 18:55:12,243 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/163c95d502b74223865c5446ca2ad0bd as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/163c95d502b74223865c5446ca2ad0bd 2023-04-18 18:55:12,246 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/163c95d502b74223865c5446ca2ad0bd, entries=150, sequenceid=50, filesize=11.6 K 2023-04-18 18:55:12,247 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for a9efbb28428802c9e45d9365ba7ee3aa in 515ms, sequenceid=50, compaction requested=true 2023-04-18 18:55:12,247 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:12,247 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:12,247 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:12,247 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:12,247 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:12,247 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:12,248 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:12,248 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): a9efbb28428802c9e45d9365ba7ee3aa/A is initiating minor compaction (all files) 2023-04-18 18:55:12,248 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:12,249 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a9efbb28428802c9e45d9365ba7ee3aa/A in TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:12,249 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): a9efbb28428802c9e45d9365ba7ee3aa/B is initiating minor compaction (all files) 2023-04-18 18:55:12,249 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9efbb28428802c9e45d9365ba7ee3aa/B in TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:12,249 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/f074f894c6a643d0841190813863f2ae, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/199c75b353ba4aeaa5100720ca0ae64d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/7b7bac1a14dc43c993bcb5a6a77eb633] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp, totalSize=82.2 K 2023-04-18 18:55:12,249 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/1a05d716fba546d6b402df3a9f5d3a22, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/af8260e1210741789f2cb61c4b00febe, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/c9ff752f5eb34ea78250f286b103a5e4] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp, totalSize=34.8 K 2023-04-18 18:55:12,249 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting f074f894c6a643d0841190813863f2ae, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681844109445 2023-04-18 18:55:12,249 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 1a05d716fba546d6b402df3a9f5d3a22, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681844109445 2023-04-18 18:55:12,249 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 199c75b353ba4aeaa5100720ca0ae64d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681844109469 2023-04-18 18:55:12,249 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting af8260e1210741789f2cb61c4b00febe, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681844109469 2023-04-18 18:55:12,250 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 7b7bac1a14dc43c993bcb5a6a77eb633, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681844111609 2023-04-18 18:55:12,250 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting c9ff752f5eb34ea78250f286b103a5e4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681844111609 2023-04-18 18:55:12,255 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): a9efbb28428802c9e45d9365ba7ee3aa#B#compaction#308 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:12,258 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): a9efbb28428802c9e45d9365ba7ee3aa#A#compaction#309 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:12,264 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:12,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:12,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:12,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:12,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:12,274 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:12,274 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:12,274 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:12,274 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:12,274 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:12,274 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:12,277 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:12,277 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:12,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:12,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:12,277 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,277 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:12,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:12,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:12,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:12,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:12,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:12,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:12,283 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:12,283 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:12,283 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:12,283 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:12,283 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:12,283 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:12,283 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:12,284 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-18 18:55:12,284 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:12,284 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:12,284 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:12,284 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:12,284 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:12,284 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:12,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:12,408 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:12,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:12,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844172438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:12,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844172445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:12,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844172546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:12,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844172547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,681 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/d9f398785f1c4dc5a25184f200c4e546 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/d9f398785f1c4dc5a25184f200c4e546 2023-04-18 18:55:12,684 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a9efbb28428802c9e45d9365ba7ee3aa/B of a9efbb28428802c9e45d9365ba7ee3aa into d9f398785f1c4dc5a25184f200c4e546(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:12,685 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:12,685 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., storeName=a9efbb28428802c9e45d9365ba7ee3aa/B, priority=13, startTime=1681844112247; duration=0sec 2023-04-18 18:55:12,685 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:12,685 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:12,686 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:12,686 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): a9efbb28428802c9e45d9365ba7ee3aa/C is initiating minor compaction (all files) 2023-04-18 18:55:12,686 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9efbb28428802c9e45d9365ba7ee3aa/C in TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:12,686 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/ccd6d5bd04a341f683195239a4de2a5a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/80cd6fe4f0c344ad8fef4fe7dc54399b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/163c95d502b74223865c5446ca2ad0bd] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp, totalSize=34.8 K 2023-04-18 18:55:12,686 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting ccd6d5bd04a341f683195239a4de2a5a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681844109445 2023-04-18 18:55:12,687 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 80cd6fe4f0c344ad8fef4fe7dc54399b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681844109469 2023-04-18 18:55:12,687 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 163c95d502b74223865c5446ca2ad0bd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681844111609 2023-04-18 18:55:12,695 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): a9efbb28428802c9e45d9365ba7ee3aa#C#compaction#311 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:12,718 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/d44a03233f8142e58aafa3eabf4fb118 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/d44a03233f8142e58aafa3eabf4fb118 2023-04-18 18:55:12,722 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418e3078a5ea06043a6b9e3ed377f8325d0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e3078a5ea06043a6b9e3ed377f8325d0 2023-04-18 18:55:12,731 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a9efbb28428802c9e45d9365ba7ee3aa/A of a9efbb28428802c9e45d9365ba7ee3aa into d44a03233f8142e58aafa3eabf4fb118(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:12,731 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:12,732 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., storeName=a9efbb28428802c9e45d9365ba7ee3aa/A, priority=13, startTime=1681844112247; duration=0sec 2023-04-18 18:55:12,732 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:12,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:12,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47258 deadline: 1681844172736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,744 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=68, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/a99ebd316403454d92eb39a6265cf586 2023-04-18 18:55:12,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:12,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47228 deadline: 1681844172739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:12,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844172751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:12,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:12,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844172752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:13,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844173059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:13,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844173060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,105 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/7809733d5fa34f918766c9bb396eac15 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/7809733d5fa34f918766c9bb396eac15 2023-04-18 18:55:13,109 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a9efbb28428802c9e45d9365ba7ee3aa/C of a9efbb28428802c9e45d9365ba7ee3aa into 7809733d5fa34f918766c9bb396eac15(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:13,109 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:13,109 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., storeName=a9efbb28428802c9e45d9365ba7ee3aa/C, priority=13, startTime=1681844112247; duration=0sec 2023-04-18 18:55:13,109 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:13,159 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/520d342c4e61461a8386f1e364a40889 2023-04-18 18:55:13,179 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/fb0212c1c6164d159968c4fbff32aff9 2023-04-18 18:55:13,183 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/a99ebd316403454d92eb39a6265cf586 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/a99ebd316403454d92eb39a6265cf586 2023-04-18 18:55:13,187 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/a99ebd316403454d92eb39a6265cf586, entries=150, sequenceid=68, filesize=25.1 K 2023-04-18 18:55:13,187 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/520d342c4e61461a8386f1e364a40889 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/520d342c4e61461a8386f1e364a40889 2023-04-18 18:55:13,191 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/520d342c4e61461a8386f1e364a40889, entries=150, sequenceid=68, filesize=11.6 K 2023-04-18 18:55:13,192 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/fb0212c1c6164d159968c4fbff32aff9 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/fb0212c1c6164d159968c4fbff32aff9 2023-04-18 18:55:13,195 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/fb0212c1c6164d159968c4fbff32aff9, entries=150, sequenceid=68, filesize=11.6 K 2023-04-18 18:55:13,195 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for a9efbb28428802c9e45d9365ba7ee3aa in 911ms, sequenceid=68, compaction requested=false 2023-04-18 18:55:13,196 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:13,196 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:13,196 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:13,196 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:13,196 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,196 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:13,196 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:13,197 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,198 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:13,198 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:13,198 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:13,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:13,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:13,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1696121[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:13,200 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:13,200 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,201 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,201 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:13,201 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:13,202 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:13,202 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,202 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:13,203 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:13,203 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:13,203 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:13,203 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:13,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:13,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:13,206 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:13,207 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:13,207 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:13,207 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:13,207 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:13,209 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,209 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:13,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,209 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:13,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:13,210 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:13,211 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:13,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,214 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:13,214 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:13,214 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:13,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:13,214 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:13,214 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:13,214 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,214 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,214 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,214 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,214 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:13,214 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:13,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:13,215 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:13,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:13,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:13,315 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:13,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:13,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:13,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:13,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:13,319 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:13,319 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:13,319 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:13,319 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:13,319 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,319 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:13,320 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:13,320 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:13,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:13,321 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,321 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:13,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:13,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:13,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:13,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:13,322 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:13,322 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:13,322 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:13,322 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:13,322 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:13,322 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:13,322 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:13,322 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:13,322 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:13,323 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-18 18:55:13,323 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:13,323 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:13,323 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:13,323 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:13,323 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:13,323 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:13,345 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041871f4d47c07cc4177960fdccc9a2fe8a9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041871f4d47c07cc4177960fdccc9a2fe8a9 2023-04-18 18:55:13,361 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=85, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/1942ed4fc7974b8ea96011a1c2bd1ad4 2023-04-18 18:55:13,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:13,572 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:13,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:13,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844173612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:13,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47242 deadline: 1681844173614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,621 DEBUG [Thread-2210] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:55:13,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:13,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844173619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:13,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844173720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:13,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844173724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:13,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47258 deadline: 1681844173747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:13,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47228 deadline: 1681844173749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,779 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/df1e6b10289d4625b67aab030ecd245a 2023-04-18 18:55:13,789 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/fbe3d2016d2a49e0890fce60229c58f4 2023-04-18 18:55:13,793 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/1942ed4fc7974b8ea96011a1c2bd1ad4 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/1942ed4fc7974b8ea96011a1c2bd1ad4 2023-04-18 18:55:13,796 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/1942ed4fc7974b8ea96011a1c2bd1ad4, entries=150, sequenceid=85, filesize=25.1 K 2023-04-18 18:55:13,797 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/df1e6b10289d4625b67aab030ecd245a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/df1e6b10289d4625b67aab030ecd245a 2023-04-18 18:55:13,801 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/df1e6b10289d4625b67aab030ecd245a, entries=150, sequenceid=85, filesize=11.6 K 2023-04-18 18:55:13,801 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/fbe3d2016d2a49e0890fce60229c58f4 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/fbe3d2016d2a49e0890fce60229c58f4 2023-04-18 18:55:13,805 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/fbe3d2016d2a49e0890fce60229c58f4, entries=150, sequenceid=85, filesize=11.6 K 2023-04-18 18:55:13,805 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for a9efbb28428802c9e45d9365ba7ee3aa in 483ms, sequenceid=85, compaction requested=true 2023-04-18 18:55:13,806 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:13,806 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:13,806 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:13,806 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:13,806 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,806 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:13,806 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:13,807 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,808 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:13,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:13,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:13,808 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,808 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:13,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:13,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:13,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:13,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:13,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69115c27[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:13,810 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:13,810 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,812 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:13,812 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:13,812 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:13,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:13,814 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,814 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:13,814 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:13,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:13,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:13,814 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:13,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:13,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:13,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:13,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:13,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:13,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:13,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:13,817 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:13,817 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:13,817 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:13,817 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:13,817 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:13,819 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,819 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:13,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:13,819 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:13,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:13,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:13,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:13,820 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:13,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:13,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:13,821 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:13,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:13,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,827 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:13,827 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:13,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:13,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:13,827 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:13,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:13,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:13,827 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:13,827 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:13,827 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:13,827 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:13,827 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:13,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:13,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:13,827 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,827 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,827 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,827 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:13,925 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-18 18:55:13,925 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:13,925 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:13,925 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:13,926 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:13,926 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:13,926 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:13,927 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:13,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:13,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:13,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:13,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:13,932 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:13,932 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:13,932 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:13,932 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:13,933 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,933 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:13,934 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:13,934 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:13,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:13,934 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,934 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:13,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:13,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:13,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:13,935 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:13,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:13,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:13,935 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:13,935 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:13,936 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:13,936 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:13,936 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:13,936 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:13,936 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:13,936 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:13,936 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:13,936 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:13,936 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:13,936 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,936 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:13,936 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:13,937 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,937 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:13,938 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,938 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:13,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:13,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b80d404[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:13,939 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:13,939 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,942 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,942 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:13,942 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:13,942 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:13,942 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,942 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:13,944 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:13,944 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:13,944 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:13,944 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:13,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:13,946 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:13,946 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:13,946 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:13,946 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:13,946 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:13,948 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,948 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:13,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,948 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:13,949 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:13,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:13,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:13,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:13,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:13,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:13,950 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,952 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,952 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:13,953 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:13,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:13,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:13,952 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:13,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:13,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:13,953 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:13,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:13,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:13,953 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:13,953 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:13,953 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,953 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,953 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:13,953 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:13,953 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:13,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:13,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844173960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:13,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:13,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844173960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,053 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:14,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:14,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:14,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:14,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:14,056 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:14,057 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:14,057 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:14,057 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:14,057 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,057 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,059 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:14,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:14,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:14,059 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,060 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,060 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:14,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:14,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,060 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:14,061 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:14,061 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:14,061 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:14,061 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:14,061 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:14,061 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:14,061 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:14,061 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:14,061 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:14,061 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:14,061 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:14,061 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:14,061 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:14,061 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,062 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:14,062 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:14,063 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,063 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:14,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:14,063 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,063 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:14,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:14,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:14,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@129cd00c[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:14,065 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:14,065 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,066 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,067 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:14,067 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:14,067 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,067 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:14,067 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:14,068 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,068 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:14,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:14,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:14,068 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:14,069 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:14,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:14,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:14,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:14,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:14,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:14,071 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:14,071 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:14,071 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:14,071 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:14,071 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:14,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:14,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844174069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,073 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,073 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,073 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:14,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:14,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:14,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844174069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:14,074 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,075 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,078 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:14,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:14,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:14,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:14,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:14,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:14,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:14,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:14,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:14,078 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:14,079 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:14,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:14,179 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:14,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:14,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:14,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:14,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:14,182 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:14,182 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:14,182 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:14,182 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:14,182 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,183 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,184 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,184 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:14,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:14,184 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,184 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:14,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:14,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:14,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:14,186 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:14,186 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:14,186 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:14,186 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:14,186 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:14,186 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:14,186 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:14,186 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:14,186 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:14,186 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:14,186 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:14,186 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,186 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:14,187 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:14,188 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,188 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:14,189 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,189 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:14,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:14,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5836db17[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:14,190 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:14,190 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,192 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,192 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:14,192 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:14,192 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,192 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:14,192 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:14,194 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:14,194 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,194 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:14,194 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:14,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:14,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:14,196 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:14,196 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:14,196 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:14,196 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:14,196 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:14,198 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,198 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,198 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:14,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:14,198 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,200 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:14,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,203 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,203 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,203 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,203 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:14,204 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:14,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:14,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:14,203 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:14,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:14,204 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:14,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:14,204 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,204 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:14,204 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,204 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:14,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:14,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:14,204 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:14,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844174274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:14,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844174276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,304 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:14,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:14,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:14,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:14,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:14,308 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:14,308 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:14,308 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:14,308 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:14,308 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,309 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,314 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:14,314 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:14,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:14,315 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,315 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:14,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:14,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:14,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:14,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:14,316 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:14,316 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:14,316 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:14,316 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:14,316 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:14,316 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:14,316 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:14,316 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:14,316 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:14,316 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:14,316 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:14,316 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,316 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:14,317 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:14,318 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,318 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:14,318 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,318 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:14,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:14,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:14,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:14,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b255452[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:14,320 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:14,320 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,321 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,321 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:14,321 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:14,321 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:14,322 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,322 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:14,324 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:14,324 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:14,324 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:14,325 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:14,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:14,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:14,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:14,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:14,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:14,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:14,327 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:14,327 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:14,327 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:14,327 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:14,327 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:14,329 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,329 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,329 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:14,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:14,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:14,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:14,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:14,330 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:14,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:14,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,331 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:14,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,333 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,333 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,334 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,334 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:14,334 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:14,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:14,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:14,334 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:14,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:14,334 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:14,334 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,334 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,334 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,334 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:14,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:14,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:14,334 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:14,346 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418b50d7c9e326c4a4f86fb2884d67b3a16 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418b50d7c9e326c4a4f86fb2884d67b3a16 2023-04-18 18:55:14,351 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=104, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/9e6076e8f60d45b4bc2941e25802d0db 2023-04-18 18:55:14,361 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/41e648254a694b8c93abd52c98dc1330 2023-04-18 18:55:14,434 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:14,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:14,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:14,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:14,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:14,438 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:14,438 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:14,438 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:14,438 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:14,438 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,438 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,440 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:14,440 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:14,440 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,440 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:14,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,441 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:14,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:14,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:14,442 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:14,442 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:14,442 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:14,442 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:14,442 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:14,442 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:14,442 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:14,442 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:14,442 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:14,442 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:14,442 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:14,442 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,442 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:14,442 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:14,445 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,445 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:14,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:14,445 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,445 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:14,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:14,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:14,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:14,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:14,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@368887c6[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:14,446 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:14,446 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,448 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,448 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:14,448 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:14,448 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:14,448 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,448 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:14,449 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,449 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:14,449 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:14,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:14,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:14,449 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:14,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:14,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:14,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:14,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:14,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:14,452 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:14,452 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:14,452 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:14,452 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:14,452 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:14,454 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,454 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:14,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,454 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:14,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:14,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:14,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:14,454 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:14,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:14,456 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:14,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,459 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,459 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,459 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,459 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:14,459 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,459 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:14,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:14,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:14,459 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:14,459 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:14,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:14,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:14,459 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,459 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:14,460 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,460 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:14,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:14,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:14,560 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:14,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:14,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:14,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:14,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:14,563 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:14,563 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:14,563 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:14,563 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:14,564 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,564 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,565 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:14,565 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:14,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:14,565 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,565 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:14,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:14,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:14,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:14,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:14,566 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:14,566 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:14,567 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:14,567 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:14,567 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:14,567 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:14,567 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:14,567 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:14,567 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:14,567 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:14,567 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:14,567 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,567 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:14,567 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:14,568 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,568 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:14,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:14,578 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,578 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:14,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:14,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:14,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:14,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:14,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42603dab[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:14,579 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:14,579 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,580 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,580 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:14,580 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:14,580 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:14,581 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,581 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:14,582 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,582 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:14,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:14,582 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:14,583 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:14,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:14,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:14,585 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:14,585 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:14,585 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:14,585 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:14,585 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:14,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:14,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844174578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,587 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,587 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,587 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:14,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:14,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:14,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:14,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:14,587 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:14,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:14,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:14,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844174584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,588 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:14,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,591 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,591 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,591 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,591 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,591 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:14,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:14,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:14,591 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,591 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:14,591 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:14,591 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:14,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:14,591 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:14,591 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:14,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:14,591 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:14,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:14,692 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:14,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:14,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:14,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:14,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:14,696 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:14,696 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:14,696 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:14,696 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:14,697 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,697 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,698 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:14,698 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:14,699 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,699 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,700 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:14,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:14,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:14,700 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:14,700 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:14,700 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:14,700 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:14,700 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:14,700 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:14,700 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:14,700 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:14,700 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:14,700 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:14,700 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:14,700 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,701 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:14,701 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:14,704 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,704 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:14,704 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,704 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:14,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:14,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ce0b893[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:14,706 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:14,706 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,710 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,710 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:14,710 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:14,710 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:14,710 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,710 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:14,711 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,712 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:14,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:14,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:14,712 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:14,712 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:14,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:14,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:14,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:14,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:14,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:14,714 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:14,714 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:14,714 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:14,714 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:14,714 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:14,716 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,716 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:14,716 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:14,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:14,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:14,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:14,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:14,717 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,718 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:14,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:14,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,720 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,720 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,720 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:14,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:14,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:14,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:14,720 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:14,720 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:14,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:14,720 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:14,720 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,720 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:14,720 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,720 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:14,720 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,720 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:14,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:14,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:14,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:14,772 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/861752b0bf214639a15f3acda0435bef 2023-04-18 18:55:14,776 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/9e6076e8f60d45b4bc2941e25802d0db as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/9e6076e8f60d45b4bc2941e25802d0db 2023-04-18 18:55:14,780 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/9e6076e8f60d45b4bc2941e25802d0db, entries=200, sequenceid=104, filesize=31.9 K 2023-04-18 18:55:14,780 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/41e648254a694b8c93abd52c98dc1330 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/41e648254a694b8c93abd52c98dc1330 2023-04-18 18:55:14,784 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/41e648254a694b8c93abd52c98dc1330, entries=150, sequenceid=104, filesize=11.6 K 2023-04-18 18:55:14,784 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/861752b0bf214639a15f3acda0435bef as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/861752b0bf214639a15f3acda0435bef 2023-04-18 18:55:14,788 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/861752b0bf214639a15f3acda0435bef, entries=150, sequenceid=104, filesize=11.6 K 2023-04-18 18:55:14,789 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for a9efbb28428802c9e45d9365ba7ee3aa in 864ms, sequenceid=104, compaction requested=true 2023-04-18 18:55:14,789 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:14,789 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:14,789 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:55:14,789 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:55:14,789 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:14,789 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:14,790 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110027 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:55:14,791 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): a9efbb28428802c9e45d9365ba7ee3aa/A is initiating minor compaction (all files) 2023-04-18 18:55:14,791 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:55:14,791 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a9efbb28428802c9e45d9365ba7ee3aa/A in TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:14,791 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): a9efbb28428802c9e45d9365ba7ee3aa/B is initiating minor compaction (all files) 2023-04-18 18:55:14,791 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9efbb28428802c9e45d9365ba7ee3aa/B in TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:14,791 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/d44a03233f8142e58aafa3eabf4fb118, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/a99ebd316403454d92eb39a6265cf586, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/1942ed4fc7974b8ea96011a1c2bd1ad4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/9e6076e8f60d45b4bc2941e25802d0db] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp, totalSize=107.4 K 2023-04-18 18:55:14,791 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/d9f398785f1c4dc5a25184f200c4e546, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/520d342c4e61461a8386f1e364a40889, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/df1e6b10289d4625b67aab030ecd245a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/41e648254a694b8c93abd52c98dc1330] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp, totalSize=46.5 K 2023-04-18 18:55:14,791 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting d44a03233f8142e58aafa3eabf4fb118, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681844111609 2023-04-18 18:55:14,791 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting d9f398785f1c4dc5a25184f200c4e546, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681844111609 2023-04-18 18:55:14,792 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting a99ebd316403454d92eb39a6265cf586, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681844111758 2023-04-18 18:55:14,793 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 520d342c4e61461a8386f1e364a40889, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681844111758 2023-04-18 18:55:14,793 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 1942ed4fc7974b8ea96011a1c2bd1ad4, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1681844112412 2023-04-18 18:55:14,793 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting df1e6b10289d4625b67aab030ecd245a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1681844112412 2023-04-18 18:55:14,794 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 9e6076e8f60d45b4bc2941e25802d0db, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681844113598 2023-04-18 18:55:14,794 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 41e648254a694b8c93abd52c98dc1330, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681844113598 2023-04-18 18:55:14,802 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): a9efbb28428802c9e45d9365ba7ee3aa#B#compaction#320 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:14,806 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): a9efbb28428802c9e45d9365ba7ee3aa#A#compaction#321 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:14,820 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:14,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:14,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:14,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:14,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:14,824 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:14,824 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:14,824 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:14,824 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:14,824 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,825 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,827 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:14,827 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:14,827 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:14,827 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:14,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:14,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:14,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:14,828 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:14,828 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:14,828 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:14,828 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:14,829 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:14,829 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:14,829 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:14,829 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 18:55:14,829 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:14,829 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:14,829 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:14,829 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:14,829 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:14,829 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:15,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:15,094 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:15,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:15,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844175143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:15,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:15,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844175147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:15,213 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/3b8aac906be345ca94072e7cceaf3075 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/3b8aac906be345ca94072e7cceaf3075 2023-04-18 18:55:15,218 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a9efbb28428802c9e45d9365ba7ee3aa/B of a9efbb28428802c9e45d9365ba7ee3aa into 3b8aac906be345ca94072e7cceaf3075(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:15,218 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:15,218 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., storeName=a9efbb28428802c9e45d9365ba7ee3aa/B, priority=12, startTime=1681844114789; duration=0sec 2023-04-18 18:55:15,218 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:15,218 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:55:15,219 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:55:15,220 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): a9efbb28428802c9e45d9365ba7ee3aa/C is initiating minor compaction (all files) 2023-04-18 18:55:15,220 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9efbb28428802c9e45d9365ba7ee3aa/C in TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:15,220 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/7809733d5fa34f918766c9bb396eac15, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/fb0212c1c6164d159968c4fbff32aff9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/fbe3d2016d2a49e0890fce60229c58f4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/861752b0bf214639a15f3acda0435bef] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp, totalSize=46.5 K 2023-04-18 18:55:15,222 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 7809733d5fa34f918766c9bb396eac15, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681844111609 2023-04-18 18:55:15,223 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting fb0212c1c6164d159968c4fbff32aff9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681844111758 2023-04-18 18:55:15,224 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting fbe3d2016d2a49e0890fce60229c58f4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1681844112412 2023-04-18 18:55:15,228 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 861752b0bf214639a15f3acda0435bef, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681844113598 2023-04-18 18:55:15,233 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/0b67a3bc8dee4444bef4cd1c1f42d191 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/0b67a3bc8dee4444bef4cd1c1f42d191 2023-04-18 18:55:15,240 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a9efbb28428802c9e45d9365ba7ee3aa/A of a9efbb28428802c9e45d9365ba7ee3aa into 0b67a3bc8dee4444bef4cd1c1f42d191(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:15,240 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:15,240 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., storeName=a9efbb28428802c9e45d9365ba7ee3aa/A, priority=12, startTime=1681844114789; duration=0sec 2023-04-18 18:55:15,240 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:15,242 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): a9efbb28428802c9e45d9365ba7ee3aa#C#compaction#323 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:15,247 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418a3485983dcdb4d59b1298e792792832a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a3485983dcdb4d59b1298e792792832a 2023-04-18 18:55:15,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:15,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844175251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:15,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:15,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844175258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:15,267 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=117, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/12e6f0cdb7024c37bdc91a38e59f749e 2023-04-18 18:55:15,268 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/8f54fbb7a1434298b47c16ad436c8513 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/8f54fbb7a1434298b47c16ad436c8513 2023-04-18 18:55:15,274 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a9efbb28428802c9e45d9365ba7ee3aa/C of a9efbb28428802c9e45d9365ba7ee3aa into 8f54fbb7a1434298b47c16ad436c8513(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:15,274 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:15,274 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., storeName=a9efbb28428802c9e45d9365ba7ee3aa/C, priority=12, startTime=1681844114789; duration=0sec 2023-04-18 18:55:15,274 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:15,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:15,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844175457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:15,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:15,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844175465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:15,684 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/1215107fcb7949c1a54a96c4d110d91a 2023-04-18 18:55:15,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:15,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844175759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:15,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:15,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47228 deadline: 1681844175759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:15,764 DEBUG [Thread-2212] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:55:15,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:15,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47258 deadline: 1681844175761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:15,769 DEBUG [Thread-2206] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4156 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:55:15,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:15,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844175771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,094 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/0a8716fbb9d1414c9ffff9846c0543ba 2023-04-18 18:55:16,099 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/12e6f0cdb7024c37bdc91a38e59f749e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/12e6f0cdb7024c37bdc91a38e59f749e 2023-04-18 18:55:16,102 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/12e6f0cdb7024c37bdc91a38e59f749e, entries=150, sequenceid=117, filesize=25.1 K 2023-04-18 18:55:16,103 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/1215107fcb7949c1a54a96c4d110d91a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/1215107fcb7949c1a54a96c4d110d91a 2023-04-18 18:55:16,106 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/1215107fcb7949c1a54a96c4d110d91a, entries=150, sequenceid=117, filesize=11.6 K 2023-04-18 18:55:16,107 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/0a8716fbb9d1414c9ffff9846c0543ba as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/0a8716fbb9d1414c9ffff9846c0543ba 2023-04-18 18:55:16,110 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/0a8716fbb9d1414c9ffff9846c0543ba, entries=150, sequenceid=117, filesize=11.6 K 2023-04-18 18:55:16,111 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for a9efbb28428802c9e45d9365ba7ee3aa in 1282ms, sequenceid=117, compaction requested=false 2023-04-18 18:55:16,111 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:16,111 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:16,111 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:16,111 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:16,111 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,111 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:16,111 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:16,113 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:16,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:16,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:16,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:16,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d76fe99[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:16,115 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:16,115 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:16,116 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:16,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:16,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:16,116 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:16,116 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:16,116 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:16,116 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,116 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:16,117 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:16,117 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:16,117 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,117 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:16,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:16,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:16,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:16,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:16,120 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:16,120 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:16,120 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:16,120 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:16,120 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:16,123 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:16,123 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:16,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:16,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:16,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:16,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:16,123 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:16,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:16,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:16,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:16,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:16,124 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:16,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:16,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:16,125 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:16,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:16,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,127 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,127 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:16,127 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:16,127 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:16,127 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:16,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:16,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:16,127 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:16,127 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:16,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:16,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:16,127 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:16,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:16,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:16,128 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:16,128 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:16,128 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:16,128 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:16,228 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:16,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:16,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:16,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:16,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:16,233 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:16,233 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:16,233 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:16,233 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:16,234 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:16,234 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:16,235 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:16,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:16,235 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:16,235 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,235 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:16,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:16,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:16,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:16,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:16,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:16,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:16,236 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:16,236 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:16,236 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:16,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:16,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:16,237 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:16,237 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:16,237 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-18 18:55:16,238 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:16,238 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:16,238 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:16,238 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:16,238 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:16,238 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:16,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:16,268 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:16,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:16,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844176291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:16,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844176297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:16,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844176402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:16,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844176404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:16,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844176605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:16,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844176610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,660 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041880ae103d51e54faca86014279bcc6f78 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041880ae103d51e54faca86014279bcc6f78 2023-04-18 18:55:16,665 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=139, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/183ad8c410334613905bd45f0d43fc97 2023-04-18 18:55:16,680 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/4e25e21f6d6f4e4abc5dc53a25dd97ce 2023-04-18 18:55:16,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:16,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844176911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:16,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:16,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844176913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,090 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/179c96d7294346b88625dc03b589b474 2023-04-18 18:55:17,094 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/183ad8c410334613905bd45f0d43fc97 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/183ad8c410334613905bd45f0d43fc97 2023-04-18 18:55:17,097 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/183ad8c410334613905bd45f0d43fc97, entries=150, sequenceid=139, filesize=25.3 K 2023-04-18 18:55:17,098 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/4e25e21f6d6f4e4abc5dc53a25dd97ce as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4e25e21f6d6f4e4abc5dc53a25dd97ce 2023-04-18 18:55:17,101 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4e25e21f6d6f4e4abc5dc53a25dd97ce, entries=150, sequenceid=139, filesize=11.8 K 2023-04-18 18:55:17,102 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/179c96d7294346b88625dc03b589b474 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/179c96d7294346b88625dc03b589b474 2023-04-18 18:55:17,106 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/179c96d7294346b88625dc03b589b474, entries=150, sequenceid=139, filesize=11.8 K 2023-04-18 18:55:17,106 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for a9efbb28428802c9e45d9365ba7ee3aa in 869ms, sequenceid=139, compaction requested=true 2023-04-18 18:55:17,106 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:17,106 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:17,107 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:17,107 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:17,107 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,107 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:17,107 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:17,108 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,108 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:17,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:17,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:17,109 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:17,109 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:17,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:17,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f32f846[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:17,110 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:17,110 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:17,112 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:17,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:17,112 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:17,112 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:17,112 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:17,112 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,112 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:17,113 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:17,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:17,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:17,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:17,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:17,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:17,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:17,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:17,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:17,116 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:17,116 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:17,116 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:17,116 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:17,116 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:17,118 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:17,118 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:17,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:17,118 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:17,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:17,118 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:17,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:17,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:17,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:17,119 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:17,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:17,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,122 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,122 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:17,122 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:17,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:17,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:17,122 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:17,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:17,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:17,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:17,122 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:17,123 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:17,123 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:17,122 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:17,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:17,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:17,123 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,123 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:17,123 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:17,123 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:17,223 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:17,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:17,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:17,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:17,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:17,226 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:17,226 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:17,226 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:17,226 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:17,227 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:17,227 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:17,229 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:17,229 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:17,230 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,230 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:17,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:17,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:17,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:17,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:17,231 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:17,231 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:17,231 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:17,231 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:17,231 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:17,231 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:17,231 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:17,231 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 18:55:17,232 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:17,232 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:17,232 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:17,232 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:17,232 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:17,232 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:17,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:17,420 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:17,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:17,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844177468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:17,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844177468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:17,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844177575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:17,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844177575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:17,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47242 deadline: 1681844177631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,637 DEBUG [Thread-2210] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:55:17,650 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418a001e25f02fe40fd8691c32345871ef6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a001e25f02fe40fd8691c32345871ef6 2023-04-18 18:55:17,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:17,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844177781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:17,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:17,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844177781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:18,055 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=152, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/36a4c7816ca84ac8bf0be74bbdc1f7e9 2023-04-18 18:55:18,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:18,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844178086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:18,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:18,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844178087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:18,466 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/4a2d065955f746d08997592d9cefb917 2023-04-18 18:55:18,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:18,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844178592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:18,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:18,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47218 deadline: 1681844178592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:18,876 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/254e0118c0cb40e2ac443b50b89c673d 2023-04-18 18:55:18,880 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/36a4c7816ca84ac8bf0be74bbdc1f7e9 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/36a4c7816ca84ac8bf0be74bbdc1f7e9 2023-04-18 18:55:18,883 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/36a4c7816ca84ac8bf0be74bbdc1f7e9, entries=150, sequenceid=152, filesize=25.3 K 2023-04-18 18:55:18,884 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/4a2d065955f746d08997592d9cefb917 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4a2d065955f746d08997592d9cefb917 2023-04-18 18:55:18,887 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4a2d065955f746d08997592d9cefb917, entries=150, sequenceid=152, filesize=11.8 K 2023-04-18 18:55:18,888 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/254e0118c0cb40e2ac443b50b89c673d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/254e0118c0cb40e2ac443b50b89c673d 2023-04-18 18:55:18,891 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/254e0118c0cb40e2ac443b50b89c673d, entries=150, sequenceid=152, filesize=11.8 K 2023-04-18 18:55:18,892 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for a9efbb28428802c9e45d9365ba7ee3aa in 1661ms, sequenceid=152, compaction requested=true 2023-04-18 18:55:18,892 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:18,892 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:18,892 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:18,892 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:18,892 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:18,892 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:18,892 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:18,894 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:18,894 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:18,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:18,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:18,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:18,894 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:18,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:18,894 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:18,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:18,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:18,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:18,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:18,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:18,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cfc096e[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:18,896 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:18,897 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:18,898 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:18,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:18,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:18,898 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:18,898 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:18,898 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:18,898 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:18,898 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:18,903 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:18,903 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:18,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:18,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:18,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:18,903 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:18,903 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:18,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:18,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:18,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:18,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:18,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:18,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:18,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:18,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:18,906 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:18,906 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:18,906 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:18,906 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:18,906 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:18,908 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:18,908 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:18,908 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:18,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:18,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:18,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:18,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:18,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:18,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:18,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:18,908 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:18,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:18,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:18,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:18,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:18,909 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:18,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:18,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:18,913 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:18,913 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:18,913 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:18,913 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:18,913 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:18,913 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:18,913 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:18,913 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:18,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:18,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:18,913 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:18,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:18,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:18,913 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:18,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:18,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:18,914 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:18,914 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:19,014 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:19,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:19,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:19,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:19,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:19,017 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:19,017 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:19,017 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:19,017 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:19,018 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:19,018 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:19,019 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:19,019 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:19,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:19,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:19,019 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:19,020 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:19,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:19,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:19,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:19,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:19,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:19,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:19,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:19,021 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:19,021 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:19,021 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:19,021 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:19,021 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:19,021 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:19,021 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:19,021 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 18:55:19,021 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:19,022 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:19,022 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:19,022 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:19,022 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:19,022 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:19,442 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041814dae8db102844b68ee2948e9098d11f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041814dae8db102844b68ee2948e9098d11f 2023-04-18 18:55:19,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:19,603 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:19,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:19,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844179624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:19,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:19,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844179635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:19,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:19,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47246 deadline: 1681844179736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:19,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:19,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844179740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:19,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:19,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47258 deadline: 1681844179792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:19,798 DEBUG [Thread-2206] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8187 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:55:19,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:19,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47228 deadline: 1681844179796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:19,804 DEBUG [Thread-2212] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8191 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:55:19,851 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=172, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/69fa5a0081274eb9896a91d2430fa235 2023-04-18 18:55:19,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:19,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844179940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:19,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:19,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844179944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:20,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844180248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:20,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844180249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,261 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/d1589ef0e83a4656817efe53f8446e78 2023-04-18 18:55:20,672 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/92d76af2363c4b83a3d0d29566720bc6 2023-04-18 18:55:20,676 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/69fa5a0081274eb9896a91d2430fa235 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/69fa5a0081274eb9896a91d2430fa235 2023-04-18 18:55:20,680 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/69fa5a0081274eb9896a91d2430fa235, entries=150, sequenceid=172, filesize=25.3 K 2023-04-18 18:55:20,681 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/d1589ef0e83a4656817efe53f8446e78 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/d1589ef0e83a4656817efe53f8446e78 2023-04-18 18:55:20,685 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/d1589ef0e83a4656817efe53f8446e78, entries=150, sequenceid=172, filesize=11.8 K 2023-04-18 18:55:20,686 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/92d76af2363c4b83a3d0d29566720bc6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/92d76af2363c4b83a3d0d29566720bc6 2023-04-18 18:55:20,689 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/92d76af2363c4b83a3d0d29566720bc6, entries=150, sequenceid=172, filesize=11.8 K 2023-04-18 18:55:20,690 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for a9efbb28428802c9e45d9365ba7ee3aa in 1669ms, sequenceid=172, compaction requested=true 2023-04-18 18:55:20,690 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:20,690 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:20,690 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:20,690 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:20,690 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,690 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:20,690 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:20,692 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,692 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:20,693 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,693 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:20,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:20,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25f11c42[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:20,694 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:20,694 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,695 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,695 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:20,695 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:20,695 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:20,695 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,695 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:20,697 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:20,697 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:20,697 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:20,697 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:20,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:20,699 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:20,699 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:20,699 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:20,699 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:20,699 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:20,702 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,702 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:20,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,702 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:20,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:20,703 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,704 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,706 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,706 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:20,706 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:20,706 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:20,706 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:20,706 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:20,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:20,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:20,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:20,706 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,707 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,707 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:20,707 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:20,707 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,707 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:20,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:20,757 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 18:55:20,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:20,757 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:20,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:20,758 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:20,758 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:20,758 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:20,807 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:20,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:20,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:20,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:20,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:20,810 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:20,810 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:20,810 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:20,810 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:20,811 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,811 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:20,812 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,812 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:20,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:20,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:20,812 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,812 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:20,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:20,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:20,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:20,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:20,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:20,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:20,813 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:20,814 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:20,814 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:20,814 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:20,814 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:20,814 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:20,814 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:20,814 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:20,814 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:20,814 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:20,814 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:20,814 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,814 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:20,814 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:20,815 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,815 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:20,816 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,816 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:20,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:20,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ff9bc53[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:20,817 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:20,817 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,819 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:20,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:20,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:20,819 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,819 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:20,820 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:20,820 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,820 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:20,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:20,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:20,821 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:20,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:20,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:20,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:20,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:20,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:20,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:20,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:20,823 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:20,823 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:20,823 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:20,823 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:20,823 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:20,824 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,824 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:20,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:20,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,824 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:20,825 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:20,826 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:20,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:20,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844180814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:20,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:20,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844180815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,828 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,828 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:20,828 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:20,828 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,828 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:20,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:20,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:20,828 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,829 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,829 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,829 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:20,829 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:20,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:20,830 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:20,830 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:20,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:20,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:20,929 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:20,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:20,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:20,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:20,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:20,932 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:20,932 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:20,932 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:20,932 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:20,933 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,933 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:20,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:20,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844180927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:20,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844180928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,935 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:20,935 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:20,936 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,936 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:20,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:20,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:20,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:20,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:20,937 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:20,937 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:20,937 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:20,938 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:20,938 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:20,938 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:20,938 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:20,938 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:20,938 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:20,938 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:20,938 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:20,938 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,938 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:20,938 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:20,939 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,939 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:20,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:20,940 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,940 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:20,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:20,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:20,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:20,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:20,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77518122[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:20,941 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:20,941 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,942 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,942 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:20,942 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:20,942 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:20,943 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,943 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:20,944 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:20,944 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:20,944 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,944 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:20,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:20,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:20,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:20,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:20,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:20,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:20,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:20,947 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:20,947 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:20,947 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:20,947 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:20,947 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:20,950 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,950 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:20,950 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,950 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:20,951 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:20,958 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,958 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:20,958 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:20,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:20,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:20,958 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:20,958 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:20,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:20,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:20,958 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:20,958 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:20,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:20,958 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,958 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:20,958 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:20,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:20,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:20,958 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:20,958 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:21,059 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:21,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:21,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:21,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:21,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:21,062 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:21,062 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:21,062 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:21,062 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:21,063 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,063 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,064 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:21,064 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:21,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:21,065 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,065 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,066 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:21,066 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:21,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:21,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:21,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:21,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:21,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:21,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:21,066 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:21,066 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:21,066 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:21,066 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:21,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:21,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:21,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:21,067 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:21,069 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,069 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,070 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,070 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:21,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b8a41df[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:21,071 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:21,071 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,072 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,072 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:21,072 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:21,072 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:21,073 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,073 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:21,074 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,074 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:21,074 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:21,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,074 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:21,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:21,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:21,076 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:21,076 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:21,076 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:21,076 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:21,076 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:21,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:21,079 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,080 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,083 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,083 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:21,083 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,083 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:21,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:21,083 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:21,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:21,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:21,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:21,083 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,083 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,083 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,083 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,084 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,084 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:21,084 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:21,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:21,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:21,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:21,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844181134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:21,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844181135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,173 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041851aacc91f62b4957966c5de7de7992a0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041851aacc91f62b4957966c5de7de7992a0 2023-04-18 18:55:21,184 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:21,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:21,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:21,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:21,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:21,187 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:21,187 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:21,187 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:21,187 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:21,188 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,188 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,189 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:21,189 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:21,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:21,190 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,190 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:21,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:21,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:21,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:21,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:21,191 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:21,191 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:21,191 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:21,191 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:21,191 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:21,191 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:21,191 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:21,191 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:21,191 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:21,191 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:21,191 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:21,191 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,191 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:21,191 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:21,192 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,192 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,193 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,193 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:21,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36b45de0[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:21,194 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:21,194 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,195 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,195 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:21,195 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,195 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:21,195 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:21,195 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:21,198 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,198 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:21,198 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,198 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:21,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:21,200 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:21,200 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:21,200 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:21,200 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:21,200 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:21,202 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,202 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,202 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:21,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:21,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:21,202 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,204 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,206 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,206 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,206 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:21,206 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:21,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:21,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:21,206 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:21,206 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:21,207 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,207 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:21,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:21,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:21,207 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,207 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:21,207 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,207 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:21,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:21,307 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:21,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:21,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:21,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:21,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:21,311 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:21,311 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:21,311 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:21,311 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:21,311 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,311 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,312 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,312 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:21,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:21,313 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,313 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:21,314 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:21,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:21,314 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:21,314 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:21,314 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:21,314 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:21,314 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:21,314 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:21,314 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:21,314 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:21,314 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:21,314 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:21,315 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:21,315 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,315 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:21,315 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:21,317 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,317 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,317 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,317 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:21,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:21,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@544d6441[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:21,319 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:21,319 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,320 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:21,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:21,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:21,320 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,321 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:21,323 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,323 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:21,323 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,323 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:21,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:21,327 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:21,327 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:21,327 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:21,327 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:21,327 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:21,329 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,329 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,329 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,329 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,331 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,341 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,341 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:21,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:21,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:21,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:21,341 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:21,341 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:21,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:21,341 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,342 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:21,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:21,341 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:21,342 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,342 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:21,342 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,342 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,342 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,442 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:21,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:21,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:21,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844181438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:21,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844181442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:21,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:21,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:21,445 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:21,445 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:21,445 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:21,445 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:21,446 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,446 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,447 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,447 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:21,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:21,447 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,447 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:21,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:21,448 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:21,448 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:21,448 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:21,448 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:21,448 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:21,448 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:21,448 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:21,448 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:21,449 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:21,449 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:21,449 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:21,449 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,449 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:21,449 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:21,450 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,450 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,450 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,450 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:21,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:21,452 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:21,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2feeb77b[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:21,452 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,453 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,453 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,453 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:21,453 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:21,453 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:21,453 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:21,460 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:21,460 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:21,460 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,460 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:21,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:21,463 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:21,463 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:21,463 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:21,463 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:21,463 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:21,464 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,464 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,464 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,465 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,466 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,468 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:21,468 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,468 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:21,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:21,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:21,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:21,468 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:21,468 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:21,469 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:21,469 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,469 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:21,469 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,469 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,469 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:21,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:21,469 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,569 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:21,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:21,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:21,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:21,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:21,572 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:21,572 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:21,572 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:21,572 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:21,573 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,573 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,574 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,574 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:21,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:21,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:21,574 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,574 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:21,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:21,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:21,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:21,575 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:21,575 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:21,575 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:21,575 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:21,575 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:21,575 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:21,575 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:21,576 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:21,576 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:21,576 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:21,576 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:21,576 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,576 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:21,576 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:21,579 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,579 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,579 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:21,580 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,580 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=185, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/7c549d28f6e84c648ab26ffd57113843 2023-04-18 18:55:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:21,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66238fe0[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:21,585 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:21,585 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,586 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,586 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:21,586 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,586 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:21,586 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:21,586 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:21,588 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,588 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:21,588 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,588 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:21,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:21,591 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:21,591 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:21,591 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:21,591 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:21,591 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:21,593 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,593 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,593 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:21,594 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,595 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,598 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,598 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:21,598 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:21,598 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:21,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:21,598 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:21,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:21,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:21,598 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,599 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,599 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,599 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,599 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,599 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:21,599 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:21,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:21,699 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:21,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:21,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:21,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:21,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:21,702 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:21,702 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:21,702 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:21,702 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:21,703 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,703 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,704 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,704 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:21,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:21,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:21,704 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,704 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:21,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:21,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:21,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:21,706 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:21,706 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:21,706 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:21,706 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:21,706 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:21,706 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:21,706 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:21,706 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:21,706 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,706 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:21,706 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:21,707 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,707 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,708 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,708 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:21,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72ec8599[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:21,709 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:21,710 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,711 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,711 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:21,711 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:21,711 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,711 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:21,711 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:21,714 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,714 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,714 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:21,714 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:21,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:21,716 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:21,716 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:21,716 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:21,716 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:21,716 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:21,718 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,719 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,719 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:21,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,719 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,722 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,724 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,724 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:21,724 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:21,724 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:21,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:21,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:21,724 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:21,724 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:21,724 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:21,724 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,724 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:21,724 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,724 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,724 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,824 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:21,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:21,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:21,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:21,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:21,828 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:21,828 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:21,828 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:21,828 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:21,828 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,828 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,832 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:21,832 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:21,833 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,833 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:21,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:21,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:21,835 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:21,835 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:21,835 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:21,835 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:21,835 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:21,835 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:21,835 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:21,835 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:21,835 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:21,835 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:21,835 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:21,835 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,835 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:21,835 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:21,837 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,837 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,837 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:21,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d3c98e9[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:21,838 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:21,839 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,840 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,840 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:21,840 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:21,840 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:21,840 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,840 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:21,841 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:21,841 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,841 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,841 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:21,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:21,843 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:21,844 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:21,844 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:21,844 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:21,844 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:21,846 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,846 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,846 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:21,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,847 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,848 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,850 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,850 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,850 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:21,850 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:21,850 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,851 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,850 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:21,851 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,851 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:21,851 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:21,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:21,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:21,852 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:21,852 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:21,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:21,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844181946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,952 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:21,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:21,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:21,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:21,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844181948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:21,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:21,955 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:21,955 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:21,955 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:21,955 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:21,955 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,956 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,957 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,957 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:21,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:21,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:21,957 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,957 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:21,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:21,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:21,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:21,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:21,958 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:21,958 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:21,958 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:21,958 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:21,958 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:21,958 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:21,958 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:21,959 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:21,959 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:21,959 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:21,959 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:21,959 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,959 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:21,959 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:21,960 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,960 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,961 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,961 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:21,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1733ab5a[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:21,962 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:21,962 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,965 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,965 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,965 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:21,965 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:21,965 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:21,965 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:21,966 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,966 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,966 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:21,966 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:21,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:21,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:21,968 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:21,969 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:21,969 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:21,969 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:21,969 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:21,970 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,970 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:21,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,970 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:21,971 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,972 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:21,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:21,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,974 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:21,974 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,974 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:21,974 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:21,974 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,974 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:21,974 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:21,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:21,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:21,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:21,974 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:21,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:21,975 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:21,975 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:21,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:21,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:21,975 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:21,975 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:21,997 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/9cd761978e3a466bb092651821edc497 2023-04-18 18:55:22,075 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:22,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:22,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:22,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:22,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:22,079 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:22,079 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:22,079 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:22,079 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:22,079 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,079 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:22,081 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:22,081 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:22,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:22,082 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,082 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:22,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:22,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:22,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:22,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:22,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:22,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:22,083 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:22,083 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:22,083 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:22,084 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:22,084 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:22,084 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:22,084 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:22,084 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:22,084 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:22,084 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:22,084 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:22,084 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,084 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:22,084 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:22,085 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,086 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:22,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:22,086 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,086 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:22,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:22,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:22,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:22,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:22,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d157ad3[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:22,087 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:22,087 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,088 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,089 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:22,089 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:22,089 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,089 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:22,089 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:22,090 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:22,090 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,090 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:22,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:22,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:22,090 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:22,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:22,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:22,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:22,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:22,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:22,093 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:22,093 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:22,093 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:22,093 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:22,093 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:22,095 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:22,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:22,095 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,095 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:22,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:22,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:22,096 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,097 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,099 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,099 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:22,099 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:22,099 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:22,099 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:22,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:22,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:22,099 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:22,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:22,099 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:22,099 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,099 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,100 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,099 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:22,100 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:22,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:22,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:22,200 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:22,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:22,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:22,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:22,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:22,203 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:22,203 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:22,203 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:22,203 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:22,204 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,204 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:22,205 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,205 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:22,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:22,205 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,205 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:22,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:22,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:22,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:22,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:22,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:22,207 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:22,207 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:22,207 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:22,207 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:22,207 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:22,207 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:22,207 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:22,207 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:22,207 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:22,207 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:22,207 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:22,207 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,207 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:22,207 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:22,212 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,212 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:22,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:22,212 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:22,212 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:22,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:22,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:22,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:22,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b43faa5[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:22,214 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:22,214 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,215 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,215 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:22,215 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:22,215 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,215 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:22,215 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:22,216 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,216 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:22,216 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:22,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,216 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:22,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:22,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:22,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:22,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:22,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:22,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:22,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:22,218 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:22,218 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:22,218 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:22,218 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:22,219 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:22,220 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,220 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,220 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:22,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:22,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,221 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,222 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,225 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,225 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:22,225 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:22,225 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:22,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:22,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:22,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:22,225 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:22,225 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:22,225 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:22,226 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,226 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:22,226 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,226 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:22,226 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:22,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:22,326 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:22,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:22,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:22,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:22,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:22,329 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:22,329 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:22,329 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:22,329 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:22,330 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,330 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:22,331 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,331 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:22,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:22,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:22,331 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,331 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:22,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:22,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:22,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:22,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:22,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:22,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:22,332 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:22,332 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:22,332 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:22,332 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:22,332 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:22,332 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:22,333 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:22,333 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:22,333 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:22,333 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:22,333 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:22,333 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,333 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:22,333 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:22,334 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,334 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:22,335 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:22,335 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:22,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:22,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:22,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:22,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5211cef4[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:22,336 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:22,336 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,337 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,337 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:22,337 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:22,337 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:22,337 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,337 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:22,340 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,340 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:22,340 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:22,340 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:22,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:22,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:22,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:22,342 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:22,342 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:22,342 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:22,342 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:22,342 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:22,344 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,344 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,344 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:22,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:22,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:22,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,344 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:22,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:22,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,345 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:22,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,347 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,347 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:22,347 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:22,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:22,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:22,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:22,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:22,348 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:22,348 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:22,347 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:22,347 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:22,348 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:22,348 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:22,348 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,348 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,348 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,411 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/bec75a94a61a466db9a9f8d062a6afbc 2023-04-18 18:55:22,415 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/7c549d28f6e84c648ab26ffd57113843 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/7c549d28f6e84c648ab26ffd57113843 2023-04-18 18:55:22,419 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/7c549d28f6e84c648ab26ffd57113843, entries=200, sequenceid=185, filesize=32.1 K 2023-04-18 18:55:22,420 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/9cd761978e3a466bb092651821edc497 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/9cd761978e3a466bb092651821edc497 2023-04-18 18:55:22,425 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/9cd761978e3a466bb092651821edc497, entries=150, sequenceid=185, filesize=11.8 K 2023-04-18 18:55:22,426 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/bec75a94a61a466db9a9f8d062a6afbc as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/bec75a94a61a466db9a9f8d062a6afbc 2023-04-18 18:55:22,430 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/bec75a94a61a466db9a9f8d062a6afbc, entries=150, sequenceid=185, filesize=11.8 K 2023-04-18 18:55:22,431 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for a9efbb28428802c9e45d9365ba7ee3aa in 1674ms, sequenceid=185, compaction requested=true 2023-04-18 18:55:22,431 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:22,431 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:22,431 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:22,431 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-18 18:55:22,431 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:22,431 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-18 18:55:22,433 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 162306 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-18 18:55:22,433 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): a9efbb28428802c9e45d9365ba7ee3aa/A is initiating minor compaction (all files) 2023-04-18 18:55:22,433 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a9efbb28428802c9e45d9365ba7ee3aa/A in TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:22,433 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/0b67a3bc8dee4444bef4cd1c1f42d191, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/12e6f0cdb7024c37bdc91a38e59f749e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/183ad8c410334613905bd45f0d43fc97, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/36a4c7816ca84ac8bf0be74bbdc1f7e9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/69fa5a0081274eb9896a91d2430fa235, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/7c549d28f6e84c648ab26ffd57113843] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp, totalSize=158.5 K 2023-04-18 18:55:22,433 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72156 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-18 18:55:22,434 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): a9efbb28428802c9e45d9365ba7ee3aa/B is initiating minor compaction (all files) 2023-04-18 18:55:22,434 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9efbb28428802c9e45d9365ba7ee3aa/B in TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:22,434 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/3b8aac906be345ca94072e7cceaf3075, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/1215107fcb7949c1a54a96c4d110d91a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4e25e21f6d6f4e4abc5dc53a25dd97ce, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4a2d065955f746d08997592d9cefb917, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/d1589ef0e83a4656817efe53f8446e78, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/9cd761978e3a466bb092651821edc497] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp, totalSize=70.5 K 2023-04-18 18:55:22,434 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 0b67a3bc8dee4444bef4cd1c1f42d191, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681844113598 2023-04-18 18:55:22,435 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 3b8aac906be345ca94072e7cceaf3075, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681844113598 2023-04-18 18:55:22,435 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 12e6f0cdb7024c37bdc91a38e59f749e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681844113950 2023-04-18 18:55:22,436 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 1215107fcb7949c1a54a96c4d110d91a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681844113950 2023-04-18 18:55:22,436 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 183ad8c410334613905bd45f0d43fc97, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1681844115119 2023-04-18 18:55:22,436 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 4e25e21f6d6f4e4abc5dc53a25dd97ce, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1681844115119 2023-04-18 18:55:22,436 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 36a4c7816ca84ac8bf0be74bbdc1f7e9, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1681844116284 2023-04-18 18:55:22,436 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 4a2d065955f746d08997592d9cefb917, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1681844116284 2023-04-18 18:55:22,436 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 69fa5a0081274eb9896a91d2430fa235, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1681844117452 2023-04-18 18:55:22,437 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting d1589ef0e83a4656817efe53f8446e78, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1681844117452 2023-04-18 18:55:22,437 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 7c549d28f6e84c648ab26ffd57113843, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1681844119618 2023-04-18 18:55:22,437 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 9cd761978e3a466bb092651821edc497, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1681844119622 2023-04-18 18:55:22,448 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:22,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:22,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:22,453 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): a9efbb28428802c9e45d9365ba7ee3aa#B#compaction#339 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:22,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:22,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:22,456 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:22,456 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:22,456 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:22,456 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:22,456 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,457 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:22,458 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:22,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:22,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:22,458 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:22,459 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,459 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:22,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:22,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:22,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:22,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:22,464 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:22,464 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:22,464 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:22,468 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:22,468 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:22,468 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:22,468 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:22,468 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-18 18:55:22,468 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:22,469 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:22,469 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:22,469 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:22,469 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:22,469 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:22,470 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): a9efbb28428802c9e45d9365ba7ee3aa#A#compaction#338 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:22,876 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/2fa99029e8fd4908972d97d50edb3e9c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/2fa99029e8fd4908972d97d50edb3e9c 2023-04-18 18:55:22,879 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a9efbb28428802c9e45d9365ba7ee3aa/B of a9efbb28428802c9e45d9365ba7ee3aa into 2fa99029e8fd4908972d97d50edb3e9c(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:22,879 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:22,879 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., storeName=a9efbb28428802c9e45d9365ba7ee3aa/B, priority=10, startTime=1681844122431; duration=0sec 2023-04-18 18:55:22,879 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:22,879 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-18 18:55:22,881 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72156 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-18 18:55:22,881 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): a9efbb28428802c9e45d9365ba7ee3aa/C is initiating minor compaction (all files) 2023-04-18 18:55:22,881 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9efbb28428802c9e45d9365ba7ee3aa/C in TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:22,881 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/8f54fbb7a1434298b47c16ad436c8513, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/0a8716fbb9d1414c9ffff9846c0543ba, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/179c96d7294346b88625dc03b589b474, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/254e0118c0cb40e2ac443b50b89c673d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/92d76af2363c4b83a3d0d29566720bc6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/bec75a94a61a466db9a9f8d062a6afbc] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp, totalSize=70.5 K 2023-04-18 18:55:22,881 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 8f54fbb7a1434298b47c16ad436c8513, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681844113598 2023-04-18 18:55:22,881 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 0a8716fbb9d1414c9ffff9846c0543ba, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681844113950 2023-04-18 18:55:22,882 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 179c96d7294346b88625dc03b589b474, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1681844115119 2023-04-18 18:55:22,882 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 254e0118c0cb40e2ac443b50b89c673d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1681844116284 2023-04-18 18:55:22,882 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 92d76af2363c4b83a3d0d29566720bc6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1681844117452 2023-04-18 18:55:22,882 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting bec75a94a61a466db9a9f8d062a6afbc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1681844119622 2023-04-18 18:55:22,892 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): a9efbb28428802c9e45d9365ba7ee3aa#C#compaction#341 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:22,898 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041830055fddff4441038ac2bcce2dbf8a58 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041830055fddff4441038ac2bcce2dbf8a58 2023-04-18 18:55:22,903 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=204, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/ae6f6e278c0c482ead6df0f76da957e3 2023-04-18 18:55:22,914 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/8cfeb418ddad4bf0a28145e66c065c2f 2023-04-18 18:55:22,922 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/5eb57c6e30a14f0e9de5d80fd1b32913 2023-04-18 18:55:22,926 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/ae6f6e278c0c482ead6df0f76da957e3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/ae6f6e278c0c482ead6df0f76da957e3 2023-04-18 18:55:22,929 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/ae6f6e278c0c482ead6df0f76da957e3, entries=150, sequenceid=204, filesize=25.3 K 2023-04-18 18:55:22,929 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/8cfeb418ddad4bf0a28145e66c065c2f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/8cfeb418ddad4bf0a28145e66c065c2f 2023-04-18 18:55:22,933 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/8cfeb418ddad4bf0a28145e66c065c2f, entries=150, sequenceid=204, filesize=11.8 K 2023-04-18 18:55:22,933 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/5eb57c6e30a14f0e9de5d80fd1b32913 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/5eb57c6e30a14f0e9de5d80fd1b32913 2023-04-18 18:55:22,937 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/5eb57c6e30a14f0e9de5d80fd1b32913, entries=150, sequenceid=204, filesize=11.8 K 2023-04-18 18:55:22,938 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for a9efbb28428802c9e45d9365ba7ee3aa in 470ms, sequenceid=204, compaction requested=false 2023-04-18 18:55:22,938 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:22,938 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:22,938 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:22,938 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:22,938 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,938 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:22,938 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:22,940 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:22,940 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:22,949 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,949 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:22,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:22,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58f2d697[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:22,950 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:22,950 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,952 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,952 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:22,952 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:22,952 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:22,952 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,952 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:22,953 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:22,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:22,954 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:22,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:22,957 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:22,957 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:22,957 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:22,957 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:22,958 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:22,959 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,959 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:22,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,959 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:22,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:22,960 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,961 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,964 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,964 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:22,964 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:22,964 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:22,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:22,964 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:22,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:22,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:22,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:22,964 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:22,965 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:22,965 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:22,964 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:22,965 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,965 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:22,965 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:22,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:23,065 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:23,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:23,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:23,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:23,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:23,068 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:23,068 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:23,068 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:23,068 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:23,069 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:23,069 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:23,070 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:23,070 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:23,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:23,070 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:23,070 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:23,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:23,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:23,071 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:23,071 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:23,071 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:23,071 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:23,071 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:23,072 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:23,072 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:23,072 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:23,072 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:23,072 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-18 18:55:23,072 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:23,072 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:23,072 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:23,072 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:23,072 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:23,072 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:23,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:23,076 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:23,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844123142, max timestamp=1681844123166Num uniques 859; 2023-04-18 18:55:23,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:55:23,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844123142, max timestamp=1681844123166Num uniques 859; 2023-04-18 18:55:23,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844123142, max timestamp=1681844123166Num uniques 859; 2023-04-18 18:55:23,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:55:23,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:55:23,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:23,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844183256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:23,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:23,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844183257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:23,301 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/a144da35e7944650a008e19e2cf68896 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/a144da35e7944650a008e19e2cf68896 2023-04-18 18:55:23,306 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a9efbb28428802c9e45d9365ba7ee3aa/C of a9efbb28428802c9e45d9365ba7ee3aa into a144da35e7944650a008e19e2cf68896(size=12.2 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:23,306 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:23,306 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., storeName=a9efbb28428802c9e45d9365ba7ee3aa/C, priority=10, startTime=1681844122431; duration=0sec 2023-04-18 18:55:23,306 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:23,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:23,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844183366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:23,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:23,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844183366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:23,495 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418210897ead31849ecb931c011f4e0b76a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418210897ead31849ecb931c011f4e0b76a 2023-04-18 18:55:23,501 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=214, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/3cbf946484294b6887d3e12cb2f9f053 2023-04-18 18:55:23,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:23,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844183572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:23,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:23,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844183573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:23,704 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/f38fe752e6cd43cf9fa66dbfb6a6cb1e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/f38fe752e6cd43cf9fa66dbfb6a6cb1e 2023-04-18 18:55:23,709 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a9efbb28428802c9e45d9365ba7ee3aa/A of a9efbb28428802c9e45d9365ba7ee3aa into f38fe752e6cd43cf9fa66dbfb6a6cb1e(size=25.7 K), total size for store is 51.0 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-18 18:55:23,709 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:23,709 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., storeName=a9efbb28428802c9e45d9365ba7ee3aa/A, priority=10, startTime=1681844122431; duration=1sec 2023-04-18 18:55:23,709 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:23,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:23,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844183882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:23,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:23,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844183882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:23,911 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/ca5667c3c8dd4775a7d9155d6113cd07 2023-04-18 18:55:24,330 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/c9314107e01b4661920e7dfd29ec566f 2023-04-18 18:55:24,334 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/3cbf946484294b6887d3e12cb2f9f053 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/3cbf946484294b6887d3e12cb2f9f053 2023-04-18 18:55:24,337 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/3cbf946484294b6887d3e12cb2f9f053, entries=100, sequenceid=214, filesize=18.5 K 2023-04-18 18:55:24,338 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/ca5667c3c8dd4775a7d9155d6113cd07 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/ca5667c3c8dd4775a7d9155d6113cd07 2023-04-18 18:55:24,341 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/ca5667c3c8dd4775a7d9155d6113cd07, entries=100, sequenceid=214, filesize=9.4 K 2023-04-18 18:55:24,342 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/c9314107e01b4661920e7dfd29ec566f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/c9314107e01b4661920e7dfd29ec566f 2023-04-18 18:55:24,345 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/c9314107e01b4661920e7dfd29ec566f, entries=100, sequenceid=214, filesize=9.4 K 2023-04-18 18:55:24,346 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for a9efbb28428802c9e45d9365ba7ee3aa in 1274ms, sequenceid=214, compaction requested=true 2023-04-18 18:55:24,346 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:24,346 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:24,346 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:24,346 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:24,346 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,346 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:24,346 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:24,347 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,347 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:24,348 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:24,348 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:24,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:24,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ef7ecb8[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:24,349 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:24,349 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:24,350 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:24,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:24,351 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,351 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:24,355 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,355 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:24,355 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:24,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,355 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:24,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:24,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:24,358 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:24,358 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:24,358 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:24,358 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:24,358 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:24,359 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:24,359 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:24,359 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:24,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:24,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:24,360 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:24,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:24,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:24,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:24,363 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:24,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:24,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,369 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:24,369 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:24,369 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,369 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:24,369 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:24,369 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,369 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:24,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:24,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:24,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:24,369 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:24,369 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:24,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:24,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:24,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:24,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:24,370 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:24,370 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:24,369 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:24,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:24,393 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=147.60 KB heapSize=420.84 KB 2023-04-18 18:55:24,393 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:24,393 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 18:55:24,393 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:24,393 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 18:55:24,394 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:24,394 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 18:55:24,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:24,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844184414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:24,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844184421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,434 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418ad6d71c3a7434171ad628227b80e1a3b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418ad6d71c3a7434171ad628227b80e1a3b 2023-04-18 18:55:24,439 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=254, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/8f5e549b31274355b3d3b32aa8479191 2023-04-18 18:55:24,448 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/fd4d20955a254160b5db2b9d78a3f1e4 2023-04-18 18:55:24,459 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/4cfa3c8d328f4673b1e404c9f46797b8 2023-04-18 18:55:24,463 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/8f5e549b31274355b3d3b32aa8479191 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/8f5e549b31274355b3d3b32aa8479191 2023-04-18 18:55:24,466 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/8f5e549b31274355b3d3b32aa8479191, entries=200, sequenceid=254, filesize=32.1 K 2023-04-18 18:55:24,467 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/fd4d20955a254160b5db2b9d78a3f1e4 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/fd4d20955a254160b5db2b9d78a3f1e4 2023-04-18 18:55:24,470 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:24,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:24,470 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/fd4d20955a254160b5db2b9d78a3f1e4, entries=150, sequenceid=254, filesize=11.8 K 2023-04-18 18:55:24,471 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/4cfa3c8d328f4673b1e404c9f46797b8 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/4cfa3c8d328f4673b1e404c9f46797b8 2023-04-18 18:55:24,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:24,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:24,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:24,473 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:24,473 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:24,473 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:24,473 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:24,473 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:24,474 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:24,475 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:24,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:24,475 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/4cfa3c8d328f4673b1e404c9f46797b8, entries=150, sequenceid=254, filesize=11.8 K 2023-04-18 18:55:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:24,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:24,476 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:24,476 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:24,476 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~420.80 KB/430896, currentSize=33.54 KB/34350 for a9efbb28428802c9e45d9365ba7ee3aa in 83ms, sequenceid=254, compaction requested=true 2023-04-18 18:55:24,476 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,476 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:24,476 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:24,476 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:24,476 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:24,476 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:24,476 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:24,476 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:24,476 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:55:24,476 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:24,477 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:55:24,476 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:24,477 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:24,477 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:24,477 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:24,477 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:24,477 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:24,477 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-18 18:55:24,477 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:24,477 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:24,478 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:24,478 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104020 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:55:24,478 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:24,478 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): a9efbb28428802c9e45d9365ba7ee3aa/A is initiating minor compaction (all files) 2023-04-18 18:55:24,478 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:24,478 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:24,478 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a9efbb28428802c9e45d9365ba7ee3aa/A in TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:24,478 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46194 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:55:24,478 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/f38fe752e6cd43cf9fa66dbfb6a6cb1e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/ae6f6e278c0c482ead6df0f76da957e3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/3cbf946484294b6887d3e12cb2f9f053, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/8f5e549b31274355b3d3b32aa8479191] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp, totalSize=101.6 K 2023-04-18 18:55:24,478 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): a9efbb28428802c9e45d9365ba7ee3aa/B is initiating minor compaction (all files) 2023-04-18 18:55:24,478 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9efbb28428802c9e45d9365ba7ee3aa/B in TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:24,478 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/2fa99029e8fd4908972d97d50edb3e9c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/8cfeb418ddad4bf0a28145e66c065c2f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/ca5667c3c8dd4775a7d9155d6113cd07, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/fd4d20955a254160b5db2b9d78a3f1e4] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp, totalSize=45.1 K 2023-04-18 18:55:24,479 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting f38fe752e6cd43cf9fa66dbfb6a6cb1e, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1681844119622 2023-04-18 18:55:24,479 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 2fa99029e8fd4908972d97d50edb3e9c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1681844119622 2023-04-18 18:55:24,479 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting ae6f6e278c0c482ead6df0f76da957e3, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681844120801 2023-04-18 18:55:24,479 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 3cbf946484294b6887d3e12cb2f9f053, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1681844123049 2023-04-18 18:55:24,479 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 8cfeb418ddad4bf0a28145e66c065c2f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681844120801 2023-04-18 18:55:24,480 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 8f5e549b31274355b3d3b32aa8479191, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1681844123204 2023-04-18 18:55:24,480 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting ca5667c3c8dd4775a7d9155d6113cd07, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1681844123049 2023-04-18 18:55:24,480 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting fd4d20955a254160b5db2b9d78a3f1e4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1681844123204 2023-04-18 18:55:24,487 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): a9efbb28428802c9e45d9365ba7ee3aa#B#compaction#352 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:24,493 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): a9efbb28428802c9e45d9365ba7ee3aa#A#compaction#351 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:24,506 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/af2aa740d72c4c86b386af6c47211121 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/af2aa740d72c4c86b386af6c47211121 2023-04-18 18:55:24,508 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304184859e93a62594f94ad38c89952445d74 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304184859e93a62594f94ad38c89952445d74 2023-04-18 18:55:24,514 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a9efbb28428802c9e45d9365ba7ee3aa/B of a9efbb28428802c9e45d9365ba7ee3aa into af2aa740d72c4c86b386af6c47211121(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:24,514 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:24,514 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., storeName=a9efbb28428802c9e45d9365ba7ee3aa/B, priority=12, startTime=1681844124476; duration=0sec 2023-04-18 18:55:24,515 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:24,515 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:55:24,516 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46194 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:55:24,516 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): a9efbb28428802c9e45d9365ba7ee3aa/C is initiating minor compaction (all files) 2023-04-18 18:55:24,516 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9efbb28428802c9e45d9365ba7ee3aa/C in TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:24,516 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/a144da35e7944650a008e19e2cf68896, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/5eb57c6e30a14f0e9de5d80fd1b32913, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/c9314107e01b4661920e7dfd29ec566f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/4cfa3c8d328f4673b1e404c9f46797b8] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp, totalSize=45.1 K 2023-04-18 18:55:24,517 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting a144da35e7944650a008e19e2cf68896, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1681844119622 2023-04-18 18:55:24,518 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 5eb57c6e30a14f0e9de5d80fd1b32913, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681844120801 2023-04-18 18:55:24,518 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=262, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/46bb4cac20124a6286de218c03d445af 2023-04-18 18:55:24,519 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting c9314107e01b4661920e7dfd29ec566f, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1681844123049 2023-04-18 18:55:24,519 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 4cfa3c8d328f4673b1e404c9f46797b8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1681844123204 2023-04-18 18:55:24,529 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): a9efbb28428802c9e45d9365ba7ee3aa#C#compaction#354 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:24,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:24,533 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:24,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844124572, max timestamp=1681844124599Num uniques 859; 2023-04-18 18:55:24,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:55:24,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844124572, max timestamp=1681844124599Num uniques 859; 2023-04-18 18:55:24,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:55:24,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:24,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844184647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:24,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844184655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:24,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844184756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:24,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844184765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,942 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/ea37117b79db411e99ba0db84e79b9f9 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/ea37117b79db411e99ba0db84e79b9f9 2023-04-18 18:55:24,946 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a9efbb28428802c9e45d9365ba7ee3aa/C of a9efbb28428802c9e45d9365ba7ee3aa into ea37117b79db411e99ba0db84e79b9f9(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:24,946 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:24,946 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., storeName=a9efbb28428802c9e45d9365ba7ee3aa/C, priority=12, startTime=1681844124477; duration=0sec 2023-04-18 18:55:24,946 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/ec9b64d68983444aa0e8efef271820cb 2023-04-18 18:55:24,946 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:24,960 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/e1083056419c418fbc2d295519fd10f4 2023-04-18 18:55:24,965 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/46bb4cac20124a6286de218c03d445af as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/46bb4cac20124a6286de218c03d445af 2023-04-18 18:55:24,969 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/46bb4cac20124a6286de218c03d445af, entries=150, sequenceid=262, filesize=25.4 K 2023-04-18 18:55:24,970 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/ec9b64d68983444aa0e8efef271820cb as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/ec9b64d68983444aa0e8efef271820cb 2023-04-18 18:55:24,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:24,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844184962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,973 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/ec9b64d68983444aa0e8efef271820cb, entries=150, sequenceid=262, filesize=11.9 K 2023-04-18 18:55:24,974 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/e1083056419c418fbc2d295519fd10f4 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/e1083056419c418fbc2d295519fd10f4 2023-04-18 18:55:24,977 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/e1083056419c418fbc2d295519fd10f4, entries=150, sequenceid=262, filesize=11.9 K 2023-04-18 18:55:24,978 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=145.36 KB/148850 for a9efbb28428802c9e45d9365ba7ee3aa in 501ms, sequenceid=262, compaction requested=false 2023-04-18 18:55:24,978 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:24,978 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:24,978 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:24,978 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:24,978 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:24,978 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=152.07 KB heapSize=443.08 KB 2023-04-18 18:55:24,979 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:24,979 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:24,979 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:24,979 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:24,979 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:24,979 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:24,980 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:24,980 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:24,982 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:24,982 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:24,983 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:24,983 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:24,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:24,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a85ab11[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:24,984 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:24,984 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:24,985 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:24,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:24,986 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:24,986 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:24,986 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:24,986 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,986 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:24,988 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:24,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:24,993 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:24,993 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:24,993 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:24,993 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:24,993 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:24,996 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:24,996 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,998 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:24,998 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:24,999 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:24,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:24,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,003 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,003 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:25,003 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:25,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:25,003 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:25,003 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:25,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:25,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:25,003 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:25,004 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,004 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,004 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,004 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:25,004 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:25,004 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:25,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:25,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844185013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,104 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:25,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:25,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:25,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:25,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:25,108 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:25,108 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:25,108 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:25,108 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:25,109 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,109 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:25,110 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:25,110 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:25,111 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,111 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:25,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:25,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:25,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:25,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:25,112 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:25,112 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:25,113 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:25,113 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:25,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:25,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:25,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:25,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:25,115 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,115 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:25,115 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,115 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:25,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:25,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@263fbcdb[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:25,119 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:25,119 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,120 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,120 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:25,121 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:25,120 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,121 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:25,121 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:25,123 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,123 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:25,123 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:25,123 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:25,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:25,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:25,125 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:25,125 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:25,125 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:25,125 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:25,125 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:25,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:25,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844185122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,127 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,127 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,127 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:25,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:25,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:25,127 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:25,129 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:25,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,130 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:25,130 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,130 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:25,131 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:25,131 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:25,131 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:25,131 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,131 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,131 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,131 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:25,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:25,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:25,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:25,131 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:25,131 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:25,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:25,231 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:25,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:25,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:25,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:25,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:25,236 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:25,236 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:25,236 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:25,236 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:25,237 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,237 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:25,238 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:25,238 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:25,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:25,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:25,239 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,239 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:25,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:25,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:25,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:25,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:25,239 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:25,239 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:25,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:25,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:25,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:25,240 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:25,240 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:25,240 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:25,240 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:25,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:25,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:25,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:25,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:25,243 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,243 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:25,243 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:25,243 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:25,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:25,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@683790e5[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:25,245 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:25,245 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,246 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,246 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:25,246 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:25,246 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:25,246 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,246 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:25,248 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,248 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:25,248 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:25,248 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:25,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:25,251 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:25,251 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:25,251 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:25,251 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:25,251 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:25,252 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,252 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,252 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:25,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:25,253 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:25,254 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:25,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:25,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,262 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:25,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:25,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:25,263 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:25,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:25,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:25,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:25,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:25,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,263 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:25,263 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:25,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:25,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:25,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:25,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844185276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,333 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/817d1a6a1bc1410eb0b8e42caf8ccf1e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/817d1a6a1bc1410eb0b8e42caf8ccf1e 2023-04-18 18:55:25,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:25,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844185329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,338 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a9efbb28428802c9e45d9365ba7ee3aa/A of a9efbb28428802c9e45d9365ba7ee3aa into 817d1a6a1bc1410eb0b8e42caf8ccf1e(size=25.9 K), total size for store is 51.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:25,339 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:25,339 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., storeName=a9efbb28428802c9e45d9365ba7ee3aa/A, priority=12, startTime=1681844124476; duration=0sec 2023-04-18 18:55:25,339 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:25,363 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:25,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:25,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:25,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:25,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:25,367 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:25,367 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:25,367 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:25,367 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:25,367 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,367 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:25,368 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,368 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:25,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:25,369 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,369 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:25,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:25,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:25,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:25,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:25,370 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:25,370 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:25,370 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:25,370 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:25,370 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:25,370 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:25,370 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:25,370 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:25,370 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:25,370 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:25,370 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:25,370 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,370 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:25,370 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:25,372 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:25,372 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:25,372 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:25,372 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:25,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:25,373 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:25,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49042377[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:25,373 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,374 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,375 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:25,375 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:25,375 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:25,375 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,375 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:25,377 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:25,377 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:25,377 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:25,377 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:25,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:25,379 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:25,379 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:25,379 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:25,379 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:25,379 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:25,381 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,381 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:25,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,381 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:25,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:25,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:25,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:25,382 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:25,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:25,384 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,386 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,386 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:25,386 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:25,386 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:25,386 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:25,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:25,386 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,386 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,386 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:25,386 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:25,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:25,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:25,386 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,386 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:25,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:25,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:25,386 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:25,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:25,394 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418583e36ad7d674a2eae40cb9976d67e6f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418583e36ad7d674a2eae40cb9976d67e6f 2023-04-18 18:55:25,398 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=299, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/bd86421651964963a5a30d8d98dfa3fc 2023-04-18 18:55:25,409 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/dd534cd76cb74611b5161d8834994fd2 2023-04-18 18:55:25,419 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=71.56 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/2cac2a66610446f6b87faf31e1ab1f1e 2023-04-18 18:55:25,423 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/bd86421651964963a5a30d8d98dfa3fc as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/bd86421651964963a5a30d8d98dfa3fc 2023-04-18 18:55:25,426 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/bd86421651964963a5a30d8d98dfa3fc, entries=200, sequenceid=299, filesize=32.3 K 2023-04-18 18:55:25,427 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/dd534cd76cb74611b5161d8834994fd2 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/dd534cd76cb74611b5161d8834994fd2 2023-04-18 18:55:25,431 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/dd534cd76cb74611b5161d8834994fd2, entries=150, sequenceid=299, filesize=11.9 K 2023-04-18 18:55:25,432 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/2cac2a66610446f6b87faf31e1ab1f1e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/2cac2a66610446f6b87faf31e1ab1f1e 2023-04-18 18:55:25,447 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/2cac2a66610446f6b87faf31e1ab1f1e, entries=150, sequenceid=299, filesize=11.9 K 2023-04-18 18:55:25,448 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~152.07 KB/155720, heapSize ~443.03 KB/453664, currentSize=26.84 KB/27480 for a9efbb28428802c9e45d9365ba7ee3aa in 470ms, sequenceid=299, compaction requested=true 2023-04-18 18:55:25,448 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:25,448 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:25,449 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:25,449 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:25,449 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:25,449 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:25,450 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85608 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:25,450 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): a9efbb28428802c9e45d9365ba7ee3aa/A is initiating minor compaction (all files) 2023-04-18 18:55:25,450 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a9efbb28428802c9e45d9365ba7ee3aa/A in TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:25,450 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/817d1a6a1bc1410eb0b8e42caf8ccf1e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/46bb4cac20124a6286de218c03d445af, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/bd86421651964963a5a30d8d98dfa3fc] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp, totalSize=83.6 K 2023-04-18 18:55:25,451 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36988 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:25,451 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): a9efbb28428802c9e45d9365ba7ee3aa/B is initiating minor compaction (all files) 2023-04-18 18:55:25,451 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9efbb28428802c9e45d9365ba7ee3aa/B in TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:25,451 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/af2aa740d72c4c86b386af6c47211121, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/ec9b64d68983444aa0e8efef271820cb, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/dd534cd76cb74611b5161d8834994fd2] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp, totalSize=36.1 K 2023-04-18 18:55:25,451 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 817d1a6a1bc1410eb0b8e42caf8ccf1e, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1681844123204 2023-04-18 18:55:25,452 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting af2aa740d72c4c86b386af6c47211121, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1681844123204 2023-04-18 18:55:25,452 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 46bb4cac20124a6286de218c03d445af, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1681844124403 2023-04-18 18:55:25,453 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting bd86421651964963a5a30d8d98dfa3fc, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681844124611 2023-04-18 18:55:25,453 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting ec9b64d68983444aa0e8efef271820cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1681844124403 2023-04-18 18:55:25,454 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting dd534cd76cb74611b5161d8834994fd2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681844124646 2023-04-18 18:55:25,478 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): a9efbb28428802c9e45d9365ba7ee3aa#B#compaction#360 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:25,485 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): a9efbb28428802c9e45d9365ba7ee3aa#A#compaction#359 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:25,487 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:25,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:25,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:25,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:25,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:25,492 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:25,492 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:25,492 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:25,492 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:25,492 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,492 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:25,494 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:25,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:25,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:25,494 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:25,494 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:25,494 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:25,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:25,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:25,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:25,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:25,497 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:25,497 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:25,497 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:25,500 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:25,500 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:25,500 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-18 18:55:25,501 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:25,501 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:25,501 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:25,501 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:25,501 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:25,501 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:25,554 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418fd6da59d0fb64850b5c09359526bb86f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418fd6da59d0fb64850b5c09359526bb86f 2023-04-18 18:55:25,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:25,747 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:25,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844125802, max timestamp=1681844125839Num uniques 859; 2023-04-18 18:55:25,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:55:25,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844125802, max timestamp=1681844125839Num uniques 859; 2023-04-18 18:55:25,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:55:25,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844125802, max timestamp=1681844125839Num uniques 859; 2023-04-18 18:55:25,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:55:25,890 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/89664d68a1e0468a8bbfe99b04c72974 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/89664d68a1e0468a8bbfe99b04c72974 2023-04-18 18:55:25,893 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a9efbb28428802c9e45d9365ba7ee3aa/B of a9efbb28428802c9e45d9365ba7ee3aa into 89664d68a1e0468a8bbfe99b04c72974(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:25,894 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:25,894 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., storeName=a9efbb28428802c9e45d9365ba7ee3aa/B, priority=13, startTime=1681844125449; duration=0sec 2023-04-18 18:55:25,894 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:25,894 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:25,895 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36988 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:25,895 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): a9efbb28428802c9e45d9365ba7ee3aa/C is initiating minor compaction (all files) 2023-04-18 18:55:25,895 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9efbb28428802c9e45d9365ba7ee3aa/C in TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:25,895 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/ea37117b79db411e99ba0db84e79b9f9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/e1083056419c418fbc2d295519fd10f4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/2cac2a66610446f6b87faf31e1ab1f1e] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp, totalSize=36.1 K 2023-04-18 18:55:25,895 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting ea37117b79db411e99ba0db84e79b9f9, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1681844123204 2023-04-18 18:55:25,896 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting e1083056419c418fbc2d295519fd10f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1681844124403 2023-04-18 18:55:25,896 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 2cac2a66610446f6b87faf31e1ab1f1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681844124646 2023-04-18 18:55:25,903 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): a9efbb28428802c9e45d9365ba7ee3aa#C#compaction#362 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:25,907 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/64a12d986c3e4b8182d269c771b9bdce as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/64a12d986c3e4b8182d269c771b9bdce 2023-04-18 18:55:25,911 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a9efbb28428802c9e45d9365ba7ee3aa/A of a9efbb28428802c9e45d9365ba7ee3aa into 64a12d986c3e4b8182d269c771b9bdce(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:25,912 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:25,912 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., storeName=a9efbb28428802c9e45d9365ba7ee3aa/A, priority=13, startTime=1681844125448; duration=0sec 2023-04-18 18:55:25,912 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:25,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681844125897, max timestamp=1681844125915Num uniques 1068; 2023-04-18 18:55:25,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-18 18:55:25,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681844125897, max timestamp=1681844125915Num uniques 1068; 2023-04-18 18:55:25,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681844125897, max timestamp=1681844125915Num uniques 1068; 2023-04-18 18:55:25,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-18 18:55:25,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-18 18:55:25,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681844125915, max timestamp=1681844125953Num uniques 1027; 2023-04-18 18:55:25,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-18 18:55:25,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681844125915, max timestamp=1681844125953Num uniques 1027; 2023-04-18 18:55:25,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681844125915, max timestamp=1681844125953Num uniques 1027; 2023-04-18 18:55:25,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-18 18:55:25,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-18 18:55:25,967 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=307, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/65533d45e4a146a4a960926fe4a5faea 2023-04-18 18:55:25,977 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/484658c8f43240dcb363ee95f43d576c 2023-04-18 18:55:26,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681844125982, max timestamp=1681844125999Num uniques 1086; 2023-04-18 18:55:26,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-18 18:55:26,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681844125982, max timestamp=1681844125999Num uniques 1086; 2023-04-18 18:55:26,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-18 18:55:26,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681844125982, max timestamp=1681844125999Num uniques 1086; 2023-04-18 18:55:26,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-18 18:55:26,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681844126033, max timestamp=1681844126088Num uniques 1095; 2023-04-18 18:55:26,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681844126033, max timestamp=1681844126088Num uniques 1095; 2023-04-18 18:55:26,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681844126033, max timestamp=1681844126088Num uniques 1095; 2023-04-18 18:55:26,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-18 18:55:26,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-18 18:55:26,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-18 18:55:26,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681844126122, max timestamp=1681844126141Num uniques 1054; 2023-04-18 18:55:26,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681844126122, max timestamp=1681844126141Num uniques 1054; 2023-04-18 18:55:26,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681844126122, max timestamp=1681844126141Num uniques 1054; 2023-04-18 18:55:26,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-18 18:55:26,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-18 18:55:26,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-18 18:55:26,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681844126168, max timestamp=1681844126198Num uniques 1063; 2023-04-18 18:55:26,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681844126168, max timestamp=1681844126198Num uniques 1063; 2023-04-18 18:55:26,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-18 18:55:26,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-18 18:55:26,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681844126168, max timestamp=1681844126198Num uniques 1063; 2023-04-18 18:55:26,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-18 18:55:26,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681844126225, max timestamp=1681844126250Num uniques 1072; 2023-04-18 18:55:26,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-18 18:55:26,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681844126225, max timestamp=1681844126250Num uniques 1072; 2023-04-18 18:55:26,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-18 18:55:26,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681844126225, max timestamp=1681844126250Num uniques 1072; 2023-04-18 18:55:26,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-18 18:55:26,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681844126276, max timestamp=1681844126290Num uniques 1031; 2023-04-18 18:55:26,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-18 18:55:26,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681844126276, max timestamp=1681844126290Num uniques 1031; 2023-04-18 18:55:26,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681844126276, max timestamp=1681844126290Num uniques 1031; 2023-04-18 18:55:26,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-18 18:55:26,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-18 18:55:26,312 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/d7e9a69548974c528809147008071756 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/d7e9a69548974c528809147008071756 2023-04-18 18:55:26,315 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a9efbb28428802c9e45d9365ba7ee3aa/C of a9efbb28428802c9e45d9365ba7ee3aa into d7e9a69548974c528809147008071756(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:26,315 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:26,315 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., storeName=a9efbb28428802c9e45d9365ba7ee3aa/C, priority=13, startTime=1681844125449; duration=0sec 2023-04-18 18:55:26,315 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:26,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681844126324, max timestamp=1681844126335Num uniques 1090; 2023-04-18 18:55:26,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-18 18:55:26,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681844126324, max timestamp=1681844126335Num uniques 1090; 2023-04-18 18:55:26,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-18 18:55:26,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681844126324, max timestamp=1681844126335Num uniques 1090; 2023-04-18 18:55:26,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-18 18:55:26,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1681844126360, max timestamp=1681844126379Num uniques 1099; 2023-04-18 18:55:26,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-04-18 18:55:26,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1681844126360, max timestamp=1681844126379Num uniques 1099; 2023-04-18 18:55:26,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-04-18 18:55:26,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1681844126360, max timestamp=1681844126379Num uniques 1099; 2023-04-18 18:55:26,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-04-18 18:55:26,388 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/e0a8b19ca87344f18bade7f28a7cf8c6 2023-04-18 18:55:26,393 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/65533d45e4a146a4a960926fe4a5faea as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/65533d45e4a146a4a960926fe4a5faea 2023-04-18 18:55:26,397 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/65533d45e4a146a4a960926fe4a5faea, entries=50, sequenceid=307, filesize=11.7 K 2023-04-18 18:55:26,397 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/484658c8f43240dcb363ee95f43d576c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/484658c8f43240dcb363ee95f43d576c 2023-04-18 18:55:26,401 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/484658c8f43240dcb363ee95f43d576c, entries=50, sequenceid=307, filesize=7.1 K 2023-04-18 18:55:26,402 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/e0a8b19ca87344f18bade7f28a7cf8c6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/e0a8b19ca87344f18bade7f28a7cf8c6 2023-04-18 18:55:26,405 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/e0a8b19ca87344f18bade7f28a7cf8c6, entries=50, sequenceid=307, filesize=7.1 K 2023-04-18 18:55:26,423 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=93.93 KB/96180 for a9efbb28428802c9e45d9365ba7ee3aa in 923ms, sequenceid=307, compaction requested=false 2023-04-18 18:55:26,423 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:26,423 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:26,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:26,423 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:26,423 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=107.34 KB heapSize=292.17 KB 2023-04-18 18:55:26,423 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:26,423 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,423 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:26,423 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:26,423 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:26,423 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:26,424 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:26,424 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:26,424 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:26,424 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:26,425 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,425 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:26,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:26,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:26,426 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,426 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:26,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:26,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:26,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:26,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e0ec465[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:26,428 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:26,428 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,429 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,429 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:26,429 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:26,429 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:26,430 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,430 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:26,435 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:26,435 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,435 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:26,435 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:26,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:26,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:26,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:26,438 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:26,438 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:26,438 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:26,438 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:26,438 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:26,441 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,441 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:26,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,441 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:26,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:26,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:26,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:26,442 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:26,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,444 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,447 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,447 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,447 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,447 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:26,447 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:26,447 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,447 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:26,447 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:26,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:26,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:26,448 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:26,448 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:26,447 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:26,448 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:26,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:26,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:26,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844186471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:26,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844186471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,548 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:26,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:26,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:26,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:26,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:26,551 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:26,551 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:26,551 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:26,551 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:26,551 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,552 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,553 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:26,553 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:26,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:26,553 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,553 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:26,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:26,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:26,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:26,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:26,555 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:26,555 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:26,555 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:26,555 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:26,555 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:26,555 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:26,555 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:26,555 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:26,555 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:26,555 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:26,555 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:26,555 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,555 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:26,555 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:26,559 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,559 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:26,560 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,560 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:26,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:26,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@450c7a1b[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:26,561 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:26,561 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,562 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,563 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:26,563 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:26,563 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:26,563 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,563 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:26,564 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:26,564 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:26,564 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,564 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:26,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:26,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:26,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:26,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:26,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:26,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:26,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:26,566 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:26,566 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:26,566 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:26,566 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:26,566 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:26,568 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,568 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,568 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:26,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:26,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:26,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:26,569 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:26,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,570 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:26,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,579 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,579 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,579 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:26,579 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:26,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:26,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:26,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:26,579 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,579 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,579 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,579 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,579 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:26,579 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:26,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:26,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:26,580 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:26,580 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:26,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:26,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844186578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:26,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844186578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,680 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:26,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:26,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:26,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:26,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:26,683 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:26,683 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:26,683 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:26,683 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:26,684 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,684 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,685 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,686 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,686 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:26,692 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:26,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:26,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:26,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:26,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:26,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:26,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:26,695 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:26,695 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:26,695 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:26,695 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:26,695 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:26,695 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:26,695 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:26,695 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:26,695 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:26,695 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:26,695 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:26,695 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,695 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:26,695 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:26,697 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,697 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:26,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:26,697 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,697 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:26,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:26,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:26,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11686ae7[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:26,699 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:26,699 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,700 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,700 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:26,700 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:26,700 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:26,700 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,700 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:26,702 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,702 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:26,702 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:26,703 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:26,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:26,705 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:26,705 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:26,705 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:26,705 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:26,705 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:26,706 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,707 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,707 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:26,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:26,707 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,709 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,710 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,710 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:26,711 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:26,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:26,711 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:26,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:26,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:26,711 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:26,711 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:26,711 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,712 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:26,712 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:26,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:26,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:26,712 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,712 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,712 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:26,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844186785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:26,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844186785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,812 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:26,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:26,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:26,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:26,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:26,815 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:26,815 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:26,815 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:26,815 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:26,816 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,816 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,817 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,817 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:26,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:26,817 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,817 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:26,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:26,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:26,818 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:26,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:26,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:26,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:26,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:26,819 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:26,819 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:26,819 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:26,819 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:26,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:26,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:26,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:26,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:26,821 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:26,821 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:26,822 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,822 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:26,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:26,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@799fec26[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:26,823 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:26,823 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,824 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,824 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:26,824 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:26,824 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,824 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:26,824 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:26,826 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:26,826 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:26,826 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,826 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:26,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:26,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:26,830 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:26,830 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:26,830 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:26,830 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:26,830 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:26,831 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,831 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:26,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,832 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:26,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:26,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:26,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:26,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:26,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,832 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,834 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:26,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:26,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:26,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:26,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:26,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:26,837 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:26,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:26,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:26,838 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:26,838 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:26,841 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418ccdbf5aaf6b742ea87956207266c551e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418ccdbf5aaf6b742ea87956207266c551e 2023-04-18 18:55:26,938 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:26,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:26,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:26,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:26,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:26,942 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:26,942 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:26,942 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:26,942 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:26,942 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,942 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,944 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:26,944 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:26,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:26,944 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,944 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:26,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:26,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:26,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:26,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:26,945 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:26,945 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:26,946 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:26,946 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:26,947 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,947 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:26,947 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,947 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:26,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:26,949 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:26,949 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11da329c[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:26,950 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,950 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:26,950 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:26,950 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,950 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:26,950 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:26,952 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:26,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:26,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:26,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:26,955 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:26,955 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:26,955 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:26,955 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:26,955 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:26,956 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,956 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,957 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:26,957 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,958 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,960 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:26,960 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:26,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:26,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:26,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:26,960 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,961 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,961 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:26,961 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:26,961 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:26,961 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:26,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:26,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:26,961 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:26,961 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,961 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:26,961 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,061 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:27,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:27,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:27,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:27,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:27,076 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:27,076 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:27,076 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:27,076 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:27,077 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,077 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,087 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:27,087 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:27,087 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,087 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:27,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,089 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:27,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:27,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:27,092 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:27,092 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:27,092 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:27,092 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:27,092 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:27,092 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:27,092 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:27,093 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:27,093 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:27,093 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:27,093 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:27,093 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,093 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:27,093 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:27,094 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:27,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:27,094 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:27,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844187090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,095 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,095 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:27,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:27,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844187090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:27,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:27,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:27,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:27,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37a05e1e[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:27,097 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:27,097 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,098 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,099 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:27,099 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:27,099 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:27,099 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,099 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:27,101 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:27,102 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:27,102 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,102 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:27,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:27,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:27,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:27,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:27,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:27,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:27,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:27,106 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:27,106 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:27,106 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:27,106 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:27,106 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:27,107 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,107 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:27,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:27,107 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:27,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:27,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:27,108 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:27,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:27,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,109 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,111 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,111 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,111 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,111 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,111 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,112 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:27,112 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:27,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:27,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:27,112 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,112 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,112 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:27,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:27,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:27,112 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:27,112 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:27,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:27,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:27,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:27,212 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:27,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:27,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:27,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:27,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:27,217 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:27,217 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:27,217 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:27,217 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:27,217 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,217 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,219 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:27,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:27,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:27,219 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,219 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,219 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:27,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:27,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:27,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:27,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:27,224 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:27,224 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:27,224 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:27,228 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:27,228 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:27,228 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:27,228 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:27,228 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:27,228 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:27,228 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:27,228 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:27,228 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,228 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:27,228 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:27,231 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,231 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:27,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:27,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:27,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:27,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:27,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:27,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65cd355f[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:27,233 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:27,233 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,234 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:27,235 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,235 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:27,236 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:27,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:27,236 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:27,236 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:27,236 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:27,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:27,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:27,238 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:27,238 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:27,238 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:27,238 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:27,238 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:27,240 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,240 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:27,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:27,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:27,241 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,241 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:27,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:27,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:27,242 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:27,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:27,247 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=512, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/06758fbed5e84335852e9d345ae5df1f 2023-04-18 18:55:27,248 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,248 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,248 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,248 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,248 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,248 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:27,248 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:27,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:27,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:27,248 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,248 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:27,248 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:27,248 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:27,249 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:27,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:27,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:27,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:27,349 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:27,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:27,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:27,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:27,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:27,352 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:27,352 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:27,352 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:27,352 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:27,353 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,353 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,354 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:27,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:27,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:27,354 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,354 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,354 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:27,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:27,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:27,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:27,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:27,355 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:27,355 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:27,355 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:27,355 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:27,356 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:27,356 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:27,356 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:27,356 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:27,356 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:27,356 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:27,356 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:27,356 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,356 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:27,356 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:27,357 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,357 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:27,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:27,358 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,358 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:27,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:27,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:27,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:27,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:27,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63351b1c[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:27,359 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:27,359 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,361 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,361 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:27,361 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:27,361 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:27,361 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,361 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:27,363 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:27,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:27,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:27,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:27,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:27,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:27,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:27,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:27,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:27,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:27,365 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:27,365 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:27,365 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:27,365 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:27,365 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:27,367 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,367 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:27,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,367 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:27,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:27,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:27,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:27,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:27,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:27,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,369 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:27,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,370 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,373 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:27,373 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,373 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:27,373 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:27,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:27,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:27,373 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:27,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:27,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:27,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:27,374 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:27,374 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:27,373 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,374 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,374 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,374 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,374 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,474 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:27,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:27,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:27,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:27,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:27,478 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:27,478 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:27,478 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:27,479 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:27,479 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,479 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,482 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,482 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:27,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:27,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:27,482 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,482 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:27,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:27,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:27,485 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:27,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:27,485 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:27,485 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:27,485 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:27,485 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:27,485 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:27,485 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:27,485 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:27,485 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:27,485 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:27,486 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:27,486 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:27,486 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,486 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:27,486 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:27,492 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,492 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:27,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:27,493 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,493 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:27,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:27,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:27,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:27,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:27,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@233b8b34[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:27,497 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:27,497 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,499 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,499 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:27,499 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:27,499 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:27,499 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,499 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:27,500 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:27,500 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:27,500 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,500 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:27,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:27,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:27,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:27,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:27,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:27,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:27,503 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:27,504 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:27,504 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:27,504 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:27,504 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:27,507 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,507 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:27,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,507 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:27,508 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,509 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:27,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,514 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,514 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,514 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,514 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,515 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:27,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:27,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:27,515 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:27,515 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:27,515 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:27,515 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:27,515 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,516 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,516 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:27,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:27,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:27,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844187598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:27,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844187600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,615 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:27,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:27,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:27,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:27,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:27,618 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:27,618 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:27,618 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:27,618 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:27,619 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,619 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,620 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:27,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:27,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:27,621 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,621 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,621 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:27,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:27,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:27,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:27,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:27,623 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:27,623 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:27,623 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:27,623 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:27,623 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:27,623 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:27,623 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:27,623 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:27,623 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:27,623 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:27,623 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:27,623 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,624 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:27,624 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:27,626 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:27,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:27,627 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,627 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:27,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:27,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:27,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:27,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@165e71e3[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:27,629 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:27,629 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,630 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,630 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:27,630 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:27,630 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:27,632 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,632 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:27,633 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:27,633 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:27,633 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:27,636 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:27,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:27,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:27,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:27,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:27,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:27,640 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:27,640 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:27,640 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:27,640 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:27,640 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:27,642 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,642 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,642 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:27,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:27,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:27,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:27,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,644 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:27,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:27,647 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,651 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,651 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,651 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:27,651 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:27,651 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:27,651 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:27,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:27,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:27,651 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,651 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,651 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,651 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:27,652 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:27,652 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:27,660 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/bed377a52de04854be853181939d44c6 2023-04-18 18:55:27,679 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/e2568054617b4c2e814f0daf18ff5b85 2023-04-18 18:55:27,685 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/06758fbed5e84335852e9d345ae5df1f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/06758fbed5e84335852e9d345ae5df1f 2023-04-18 18:55:27,703 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/06758fbed5e84335852e9d345ae5df1f, entries=200, sequenceid=512, filesize=32.3 K 2023-04-18 18:55:27,704 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/bed377a52de04854be853181939d44c6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/bed377a52de04854be853181939d44c6 2023-04-18 18:55:27,708 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/bed377a52de04854be853181939d44c6, entries=150, sequenceid=512, filesize=11.9 K 2023-04-18 18:55:27,709 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/e2568054617b4c2e814f0daf18ff5b85 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/e2568054617b4c2e814f0daf18ff5b85 2023-04-18 18:55:27,712 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/e2568054617b4c2e814f0daf18ff5b85, entries=150, sequenceid=512, filesize=11.9 K 2023-04-18 18:55:27,714 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~292.13 KB/299136, currentSize=73.80 KB/75570 for a9efbb28428802c9e45d9365ba7ee3aa in 1290ms, sequenceid=512, compaction requested=true 2023-04-18 18:55:27,714 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:27,714 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:27,714 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:27,714 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:27,714 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:27,714 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:27,715 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 71770 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:27,715 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): a9efbb28428802c9e45d9365ba7ee3aa/A is initiating minor compaction (all files) 2023-04-18 18:55:27,715 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a9efbb28428802c9e45d9365ba7ee3aa/A in TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:27,715 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 32354 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:27,715 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/64a12d986c3e4b8182d269c771b9bdce, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/65533d45e4a146a4a960926fe4a5faea, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/06758fbed5e84335852e9d345ae5df1f] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp, totalSize=70.1 K 2023-04-18 18:55:27,715 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): a9efbb28428802c9e45d9365ba7ee3aa/B is initiating minor compaction (all files) 2023-04-18 18:55:27,715 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9efbb28428802c9e45d9365ba7ee3aa/B in TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:27,715 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/89664d68a1e0468a8bbfe99b04c72974, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/484658c8f43240dcb363ee95f43d576c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/bed377a52de04854be853181939d44c6] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp, totalSize=31.6 K 2023-04-18 18:55:27,716 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 64a12d986c3e4b8182d269c771b9bdce, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681844124646 2023-04-18 18:55:27,716 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 89664d68a1e0468a8bbfe99b04c72974, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681844124646 2023-04-18 18:55:27,716 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 65533d45e4a146a4a960926fe4a5faea, keycount=50, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1681844125011 2023-04-18 18:55:27,716 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 484658c8f43240dcb363ee95f43d576c, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1681844125011 2023-04-18 18:55:27,717 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting bed377a52de04854be853181939d44c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1681844126392 2023-04-18 18:55:27,717 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 06758fbed5e84335852e9d345ae5df1f, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1681844126392 2023-04-18 18:55:27,723 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): a9efbb28428802c9e45d9365ba7ee3aa#B#compaction#368 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:27,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:27,727 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-18 18:55:27,727 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:27,727 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:27,727 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:27,727 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:27,727 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:27,727 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:27,730 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): a9efbb28428802c9e45d9365ba7ee3aa#A#compaction#369 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:27,743 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/4d664e6b3fa14c189c2d62ad5e7cd313 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4d664e6b3fa14c189c2d62ad5e7cd313 2023-04-18 18:55:27,748 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418eca83accecec4acda86621899c5dea12 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418eca83accecec4acda86621899c5dea12 2023-04-18 18:55:27,749 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a9efbb28428802c9e45d9365ba7ee3aa/B of a9efbb28428802c9e45d9365ba7ee3aa into 4d664e6b3fa14c189c2d62ad5e7cd313(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:27,749 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:27,749 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., storeName=a9efbb28428802c9e45d9365ba7ee3aa/B, priority=13, startTime=1681844127714; duration=0sec 2023-04-18 18:55:27,749 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:27,749 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:27,751 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 32354 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:27,751 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): a9efbb28428802c9e45d9365ba7ee3aa/C is initiating minor compaction (all files) 2023-04-18 18:55:27,751 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9efbb28428802c9e45d9365ba7ee3aa/C in TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:27,751 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/d7e9a69548974c528809147008071756, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/e0a8b19ca87344f18bade7f28a7cf8c6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/e2568054617b4c2e814f0daf18ff5b85] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp, totalSize=31.6 K 2023-04-18 18:55:27,756 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting d7e9a69548974c528809147008071756, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681844124646 2023-04-18 18:55:27,756 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=527, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/bfc743b7bc1d46668e64efa179b06a31 2023-04-18 18:55:27,757 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting e0a8b19ca87344f18bade7f28a7cf8c6, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1681844125011 2023-04-18 18:55:27,757 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting e2568054617b4c2e814f0daf18ff5b85, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1681844126392 2023-04-18 18:55:27,757 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:27,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:27,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:27,762 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/c4c4ecaac5dd4ccd9abc2d7a5c13f6ef as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/c4c4ecaac5dd4ccd9abc2d7a5c13f6ef 2023-04-18 18:55:27,777 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): a9efbb28428802c9e45d9365ba7ee3aa#C#compaction#371 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:27,778 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a9efbb28428802c9e45d9365ba7ee3aa/A of a9efbb28428802c9e45d9365ba7ee3aa into c4c4ecaac5dd4ccd9abc2d7a5c13f6ef(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:27,778 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:27,778 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., storeName=a9efbb28428802c9e45d9365ba7ee3aa/A, priority=13, startTime=1681844127714; duration=0sec 2023-04-18 18:55:27,778 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:27,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:27,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:27,784 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:27,784 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:27,784 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:27,784 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:27,785 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,785 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,786 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,786 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:27,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:27,786 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,786 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:27,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:27,793 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:27,793 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:27,793 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:27,794 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:27,794 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:27,794 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:27,795 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:27,795 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:27,795 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:27,795 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:27,795 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:27,795 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:27,798 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,798 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:27,798 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,798 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:27,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:27,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a35d9ba[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:27,805 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:27,805 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,807 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,807 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:27,807 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:27,807 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:27,808 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,808 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:27,810 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,810 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:27,810 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:27,810 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:27,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:27,813 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:27,813 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:27,813 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:27,813 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:27,813 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:27,814 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,814 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:27,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,814 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:27,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:27,815 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:27,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,816 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,819 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:27,819 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:27,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:27,819 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,819 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,819 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,819 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,819 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,819 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,819 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:27,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:27,819 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:27,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:27,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:27,820 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:27,820 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:27,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:27,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47242 deadline: 1681844187842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,920 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:27,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:27,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:27,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:27,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:27,924 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:27,924 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:27,924 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:27,924 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:27,924 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,925 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,926 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,926 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:27,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:27,926 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,926 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:27,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:27,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:27,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:27,927 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:27,927 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:27,927 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:27,927 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:27,927 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:27,927 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:27,927 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:27,927 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:27,928 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:27,928 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:27,928 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:27,928 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,928 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:27,928 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:27,930 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,930 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:27,930 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,930 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:27,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:27,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5113cc1b[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:27,932 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:27,932 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,934 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,934 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:27,935 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,935 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:27,935 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:27,935 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:27,939 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,939 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:27,939 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:27,939 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:27,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:27,945 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:27,945 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:27,945 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:27,945 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:27,945 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:27,946 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,946 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,946 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:27,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:27,949 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,950 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:27,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:27,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47242 deadline: 1681844187947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,952 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,952 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,952 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:27,952 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:27,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:27,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:27,952 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:27,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:27,952 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:27,952 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:27,952 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:27,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:27,952 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:27,952 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:27,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:27,953 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:27,953 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,052 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:28,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:28,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:28,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:28,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:28,057 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:28,057 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:28,058 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:28,058 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:28,058 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,058 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,060 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:28,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:28,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:28,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,061 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,062 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,062 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:28,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:28,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:28,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:28,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:28,063 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:28,063 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:28,063 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:28,063 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:28,063 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:28,063 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:28,063 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:28,063 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:28,063 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:28,063 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:28,063 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:28,063 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,063 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:28,064 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:28,065 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:28,065 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:28,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:28,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:28,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e8e9f26[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:28,067 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:28,067 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,069 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,069 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:28,069 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:28,069 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:28,069 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,069 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:28,071 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:28,071 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:28,071 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:28,072 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:28,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:28,077 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:28,077 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:28,077 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:28,077 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:28,077 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:28,080 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,080 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,080 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:28,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:28,081 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:28,082 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,084 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,084 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,084 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,084 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,084 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,084 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,084 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:28,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:28,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:28,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:28,084 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:28,084 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:28,084 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,085 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:28,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:28,085 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:28,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:28,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:28,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47242 deadline: 1681844188152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,185 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:28,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:28,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:28,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:28,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:28,188 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:28,188 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:28,188 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:28,188 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:28,188 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,189 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,190 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,190 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:28,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:28,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:28,190 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,190 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:28,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:28,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:28,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:28,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:28,191 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:28,191 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:28,191 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:28,191 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:28,191 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:28,191 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:28,192 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:28,192 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:28,192 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:28,192 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:28,192 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:28,192 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,192 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:28,192 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:28,194 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,194 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:28,194 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,194 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:28,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:28,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bba227d[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:28,195 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:28,195 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,196 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,197 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:28,197 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:28,197 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:28,199 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:28,199 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:28,199 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,199 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:28,201 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/477c5624ebc0488ca452ca77c70a6daa as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/477c5624ebc0488ca452ca77c70a6daa 2023-04-18 18:55:28,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,202 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,202 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,203 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/fefe96a11cd14043b250fb2a16202b3d 2023-04-18 18:55:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:28,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:28,204 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:28,204 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:28,204 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:28,204 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:28,204 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:28,205 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a9efbb28428802c9e45d9365ba7ee3aa/C of a9efbb28428802c9e45d9365ba7ee3aa into 477c5624ebc0488ca452ca77c70a6daa(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:28,205 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:28,205 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., storeName=a9efbb28428802c9e45d9365ba7ee3aa/C, priority=13, startTime=1681844127714; duration=0sec 2023-04-18 18:55:28,206 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:28,206 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,206 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:28,206 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:28,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:28,207 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,210 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,212 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,212 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,213 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,213 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,213 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,213 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:28,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:28,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:28,213 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:28,213 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:28,213 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:28,213 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:28,213 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:28,213 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:28,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:28,313 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:28,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:28,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:28,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:28,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:28,317 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:28,317 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:28,317 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:28,317 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:28,317 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,317 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,318 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,318 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:28,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:28,319 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,319 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:28,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:28,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:28,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:28,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:28,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:28,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:28,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:28,320 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:28,320 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:28,320 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:28,320 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:28,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:28,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:28,321 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,321 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:28,321 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:28,323 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,323 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:28,323 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,323 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:28,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:28,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ff6cab9[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:28,325 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:28,325 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,326 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,326 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:28,326 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:28,326 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:28,327 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,327 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:28,328 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:28,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:28,331 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:28,331 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:28,331 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:28,331 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:28,331 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:28,340 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:28,340 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,341 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:28,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:28,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:28,341 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,344 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:28,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,348 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:28,349 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:28,349 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:28,349 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,349 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,349 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,349 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:28,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:28,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:28,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:28,349 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,349 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,349 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,349 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:28,349 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:28,449 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:28,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:28,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:28,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:28,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:28,453 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:28,453 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:28,453 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:28,453 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:28,453 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,453 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,455 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,455 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:28,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:28,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:28,455 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,455 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:28,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:28,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:28,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:28,456 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:28,456 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:28,456 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:28,456 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:28,456 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:28,456 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:28,456 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:28,456 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:28,457 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:28,457 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:28,457 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:28,457 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,457 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:28,457 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:28,460 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,460 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:28,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:28,460 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,460 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:28,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:28,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:28,462 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:28,462 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30caa248[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:28,463 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:28,463 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:28,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47242 deadline: 1681844188458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,463 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:28,463 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,463 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:28,463 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:28,465 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:28,465 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,465 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:28,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:28,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:28,465 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:28,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:28,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:28,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:28,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:28,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:28,467 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:28,467 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:28,467 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:28,467 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:28,467 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:28,469 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,469 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:28,469 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:28,469 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:28,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:28,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:28,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,471 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:28,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,475 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:28,475 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:28,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:28,475 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:28,475 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,476 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,476 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,476 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,476 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,476 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,476 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:28,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:28,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:28,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:28,476 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:28,476 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:28,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:28,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:28,576 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:28,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:28,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:28,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:28,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:28,579 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:28,579 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:28,579 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:28,579 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:28,580 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,580 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,581 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:28,581 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:28,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:28,581 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,581 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:28,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:28,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:28,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:28,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:28,582 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:28,582 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:28,582 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:28,583 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:28,583 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:28,583 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:28,583 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:28,583 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:28,583 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:28,583 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:28,583 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:28,583 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,583 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:28,583 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:28,584 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,584 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:28,585 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,585 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:28,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:28,586 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:28,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ae77cd8[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:28,586 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,587 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,587 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,587 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:28,587 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:28,587 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:28,587 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:28,590 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,590 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:28,590 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:28,590 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:28,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:28,592 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:28,592 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:28,592 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:28,592 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:28,592 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:28,594 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,594 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:28,594 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:28,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:28,594 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,595 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,597 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,597 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,597 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,597 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,597 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:28,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:28,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:28,597 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:28,597 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,597 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:28,597 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:28,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:28,598 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,598 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:28,598 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:28,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:28,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:28,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47246 deadline: 1681844188606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:28,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47218 deadline: 1681844188611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,618 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/efcc8bdba4454419b3f73c548693df0d 2023-04-18 18:55:28,621 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/bfc743b7bc1d46668e64efa179b06a31 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/bfc743b7bc1d46668e64efa179b06a31 2023-04-18 18:55:28,624 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/bfc743b7bc1d46668e64efa179b06a31, entries=200, sequenceid=527, filesize=32.3 K 2023-04-18 18:55:28,625 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/fefe96a11cd14043b250fb2a16202b3d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/fefe96a11cd14043b250fb2a16202b3d 2023-04-18 18:55:28,628 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/fefe96a11cd14043b250fb2a16202b3d, entries=150, sequenceid=527, filesize=11.9 K 2023-04-18 18:55:28,629 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/efcc8bdba4454419b3f73c548693df0d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/efcc8bdba4454419b3f73c548693df0d 2023-04-18 18:55:28,633 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/efcc8bdba4454419b3f73c548693df0d, entries=150, sequenceid=527, filesize=11.9 K 2023-04-18 18:55:28,633 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for a9efbb28428802c9e45d9365ba7ee3aa in 906ms, sequenceid=527, compaction requested=false 2023-04-18 18:55:28,633 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:28,698 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:28,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:28,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:28,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:28,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:28,702 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:28,702 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:28,703 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:28,703 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:28,703 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,703 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,704 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,704 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:28,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:28,704 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:28,704 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:28,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:28,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:28,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:28,705 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:28,706 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:28,706 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:28,706 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:28,706 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:28,706 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:28,706 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. started... 2023-04-18 18:55:28,706 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-18 18:55:28,706 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:28,706 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:28,706 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:28,706 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:28,707 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:28,707 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:28,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:28,972 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. as already flushing 2023-04-18 18:55:29,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:29,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47242 deadline: 1681844189015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:29,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:29,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47242 deadline: 1681844189121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:29,125 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304187d8b14af533742679fffda1d576acc64 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304187d8b14af533742679fffda1d576acc64 2023-04-18 18:55:29,130 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=547, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/0daad943a3ff47e2b6e3c5fd701d5377 2023-04-18 18:55:29,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:29,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47242 deadline: 1681844189327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:29,425 DEBUG [Thread-2221] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2abd7961 to 127.0.0.1:65091 2023-04-18 18:55:29,425 DEBUG [Thread-2221] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:29,430 DEBUG [Thread-2223] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x660630e8 to 127.0.0.1:65091 2023-04-18 18:55:29,430 DEBUG [Thread-2223] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:29,432 DEBUG [Thread-2219] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ea43962 to 127.0.0.1:65091 2023-04-18 18:55:29,432 DEBUG [Thread-2219] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:29,434 DEBUG [Thread-2217] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6846dddb to 127.0.0.1:65091 2023-04-18 18:55:29,434 DEBUG [Thread-2217] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:29,437 DEBUG [Thread-2225] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x793239ee to 127.0.0.1:65091 2023-04-18 18:55:29,437 DEBUG [Thread-2225] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:29,539 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/4252f9d772744adf937a156f395734a4 2023-04-18 18:55:29,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:29,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47242 deadline: 1681844189633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:29,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:29,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47228 deadline: 1681844189833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:29,835 DEBUG [Thread-2212] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18222 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:55:29,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:29,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47258 deadline: 1681844189838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:29,839 DEBUG [Thread-2206] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18228 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:55:29,948 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/6d15867aa3044893a8d61a191237d5d1 2023-04-18 18:55:29,951 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/0daad943a3ff47e2b6e3c5fd701d5377 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/0daad943a3ff47e2b6e3c5fd701d5377 2023-04-18 18:55:29,954 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/0daad943a3ff47e2b6e3c5fd701d5377, entries=150, sequenceid=547, filesize=25.4 K 2023-04-18 18:55:29,955 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/4252f9d772744adf937a156f395734a4 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4252f9d772744adf937a156f395734a4 2023-04-18 18:55:29,958 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4252f9d772744adf937a156f395734a4, entries=150, sequenceid=547, filesize=11.9 K 2023-04-18 18:55:29,958 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/6d15867aa3044893a8d61a191237d5d1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/6d15867aa3044893a8d61a191237d5d1 2023-04-18 18:55:29,961 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/6d15867aa3044893a8d61a191237d5d1, entries=150, sequenceid=547, filesize=11.9 K 2023-04-18 18:55:29,961 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for a9efbb28428802c9e45d9365ba7ee3aa in 1255ms, sequenceid=547, compaction requested=true 2023-04-18 18:55:29,962 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:29,962 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:29,962 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:29,962 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:29,962 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:29,962 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:29,962 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:29,963 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:29,963 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:29,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:29,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:29,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:29,963 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:29,963 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:29,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:29,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:29,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:29,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:29,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:29,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:29,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70541233[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:29,965 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:29,965 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:29,966 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:29,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:29,966 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:29,966 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:29,966 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:29,966 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:29,966 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:29,967 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:29,967 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:29,967 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:29,967 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:29,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:29,969 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:29,969 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:29,969 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:29,969 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:29,969 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:29,972 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:29,972 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:29,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:29,972 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:29,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:29,972 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:29,973 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:29,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:29,975 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:29,975 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:29,975 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:29,975 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:29,975 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:29,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:29,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:29,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:29,975 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:29,975 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:29,975 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:29,975 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:29,975 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:29,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:29,975 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:29,975 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:30,076 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:30,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:30,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:30,136 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-18 18:55:30,136 DEBUG [Thread-2210] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5b7d4a51 to 127.0.0.1:65091 2023-04-18 18:55:30,136 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:30,136 DEBUG [Thread-2210] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:30,136 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:30,136 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:30,136 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:30,136 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:30,136 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:30,147 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041864a2c80975ca4e3d8721ebb61c2c245d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041864a2c80975ca4e3d8721ebb61c2c245d 2023-04-18 18:55:30,551 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=563, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/31cfdfb385a443fbb6f30b16443c4758 2023-04-18 18:55:30,624 DEBUG [Thread-2208] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6e7e18ef to 127.0.0.1:65091 2023-04-18 18:55:30,624 DEBUG [Thread-2214] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6bbc9fc8 to 127.0.0.1:65091 2023-04-18 18:55:30,624 DEBUG [Thread-2208] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:30,624 DEBUG [Thread-2214] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:30,960 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=563 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/bfdc915a3ee94894805bde249af6c7e7 2023-04-18 18:55:31,368 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=563 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/45db1eb210824b9283af200075ed39d3 2023-04-18 18:55:31,372 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/31cfdfb385a443fbb6f30b16443c4758 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/31cfdfb385a443fbb6f30b16443c4758 2023-04-18 18:55:31,374 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/31cfdfb385a443fbb6f30b16443c4758, entries=150, sequenceid=563, filesize=25.4 K 2023-04-18 18:55:31,375 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/bfdc915a3ee94894805bde249af6c7e7 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/bfdc915a3ee94894805bde249af6c7e7 2023-04-18 18:55:31,378 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/bfdc915a3ee94894805bde249af6c7e7, entries=150, sequenceid=563, filesize=11.9 K 2023-04-18 18:55:31,378 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/45db1eb210824b9283af200075ed39d3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/45db1eb210824b9283af200075ed39d3 2023-04-18 18:55:31,381 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/45db1eb210824b9283af200075ed39d3, entries=150, sequenceid=563, filesize=11.9 K 2023-04-18 18:55:31,382 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=13.42 KB/13740 for a9efbb28428802c9e45d9365ba7ee3aa in 1246ms, sequenceid=563, compaction requested=true 2023-04-18 18:55:31,382 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:31,382 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:31,382 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:31,382 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:55:31,382 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:55:31,382 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:31,383 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 112008 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:55:31,383 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:55:31,383 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): a9efbb28428802c9e45d9365ba7ee3aa/A is initiating minor compaction (all files) 2023-04-18 18:55:31,383 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): a9efbb28428802c9e45d9365ba7ee3aa/B is initiating minor compaction (all files) 2023-04-18 18:55:31,383 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a9efbb28428802c9e45d9365ba7ee3aa/A in TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:31,383 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9efbb28428802c9e45d9365ba7ee3aa/B in TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:31,383 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/c4c4ecaac5dd4ccd9abc2d7a5c13f6ef, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/bfc743b7bc1d46668e64efa179b06a31, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/0daad943a3ff47e2b6e3c5fd701d5377, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/31cfdfb385a443fbb6f30b16443c4758] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp, totalSize=109.4 K 2023-04-18 18:55:31,383 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4d664e6b3fa14c189c2d62ad5e7cd313, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/fefe96a11cd14043b250fb2a16202b3d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4252f9d772744adf937a156f395734a4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/bfdc915a3ee94894805bde249af6c7e7] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp, totalSize=48.4 K 2023-04-18 18:55:31,384 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting c4c4ecaac5dd4ccd9abc2d7a5c13f6ef, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1681844126392 2023-04-18 18:55:31,384 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 4d664e6b3fa14c189c2d62ad5e7cd313, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1681844126392 2023-04-18 18:55:31,384 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting bfc743b7bc1d46668e64efa179b06a31, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1681844126449 2023-04-18 18:55:31,384 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting fefe96a11cd14043b250fb2a16202b3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1681844126454 2023-04-18 18:55:31,384 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 0daad943a3ff47e2b6e3c5fd701d5377, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1681844127810 2023-04-18 18:55:31,384 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 4252f9d772744adf937a156f395734a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1681844127810 2023-04-18 18:55:31,384 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 31cfdfb385a443fbb6f30b16443c4758, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1681844128980 2023-04-18 18:55:31,384 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting bfdc915a3ee94894805bde249af6c7e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1681844128980 2023-04-18 18:55:31,391 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): a9efbb28428802c9e45d9365ba7ee3aa#B#compaction#381 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:31,393 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): a9efbb28428802c9e45d9365ba7ee3aa#A#compaction#380 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:31,798 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/74380f3a4d97444f9909c9e75be3e14a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/74380f3a4d97444f9909c9e75be3e14a 2023-04-18 18:55:31,802 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a9efbb28428802c9e45d9365ba7ee3aa/B of a9efbb28428802c9e45d9365ba7ee3aa into 74380f3a4d97444f9909c9e75be3e14a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:31,802 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:31,802 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., storeName=a9efbb28428802c9e45d9365ba7ee3aa/B, priority=12, startTime=1681844131382; duration=0sec 2023-04-18 18:55:31,803 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:31,803 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:55:31,803 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:55:31,804 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): a9efbb28428802c9e45d9365ba7ee3aa/C is initiating minor compaction (all files) 2023-04-18 18:55:31,804 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9efbb28428802c9e45d9365ba7ee3aa/C in TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:31,804 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/477c5624ebc0488ca452ca77c70a6daa, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/efcc8bdba4454419b3f73c548693df0d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/6d15867aa3044893a8d61a191237d5d1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/45db1eb210824b9283af200075ed39d3] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp, totalSize=48.4 K 2023-04-18 18:55:31,804 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 477c5624ebc0488ca452ca77c70a6daa, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1681844126392 2023-04-18 18:55:31,804 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting efcc8bdba4454419b3f73c548693df0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1681844126454 2023-04-18 18:55:31,805 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 6d15867aa3044893a8d61a191237d5d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1681844127810 2023-04-18 18:55:31,805 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 45db1eb210824b9283af200075ed39d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1681844128980 2023-04-18 18:55:31,811 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): a9efbb28428802c9e45d9365ba7ee3aa#C#compaction#382 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:32,218 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/0c17e26348e2441993714f0aed5a3196 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/0c17e26348e2441993714f0aed5a3196 2023-04-18 18:55:32,222 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a9efbb28428802c9e45d9365ba7ee3aa/C of a9efbb28428802c9e45d9365ba7ee3aa into 0c17e26348e2441993714f0aed5a3196(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:32,222 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:32,222 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., storeName=a9efbb28428802c9e45d9365ba7ee3aa/C, priority=12, startTime=1681844131382; duration=0sec 2023-04-18 18:55:32,222 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:32,610 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/516541e819bb43e0b865097342055b37 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/516541e819bb43e0b865097342055b37 2023-04-18 18:55:32,615 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a9efbb28428802c9e45d9365ba7ee3aa/A of a9efbb28428802c9e45d9365ba7ee3aa into 516541e819bb43e0b865097342055b37(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-18 18:55:32,615 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:32,615 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa., storeName=a9efbb28428802c9e45d9365ba7ee3aa/A, priority=12, startTime=1681844131382; duration=1sec 2023-04-18 18:55:32,615 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:39,841 DEBUG [Thread-2212] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4335c1f2 to 127.0.0.1:65091 2023-04-18 18:55:39,841 DEBUG [Thread-2212] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:39,940 DEBUG [Thread-2206] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2deaf1fa to 127.0.0.1:65091 2023-04-18 18:55:39,940 DEBUG [Thread-2206] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:39,940 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-18 18:55:39,940 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 8 2023-04-18 18:55:39,940 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 218 2023-04-18 18:55:39,940 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 33 2023-04-18 18:55:39,940 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 9 2023-04-18 18:55:39,940 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 214 2023-04-18 18:55:39,940 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-18 18:55:39,940 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-18 18:55:39,940 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(402): scanned 1862 2023-04-18 18:55:39,941 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(403): verified 5586 rows 2023-04-18 18:55:39,941 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(402): scanned 1851 2023-04-18 18:55:39,941 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(403): verified 5553 rows 2023-04-18 18:55:39,941 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(402): scanned 1864 2023-04-18 18:55:39,941 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(403): verified 5592 rows 2023-04-18 18:55:39,941 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(402): scanned 1862 2023-04-18 18:55:39,941 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(403): verified 5586 rows 2023-04-18 18:55:39,941 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(402): scanned 1855 2023-04-18 18:55:39,941 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(403): verified 5565 rows 2023-04-18 18:55:39,941 INFO [Listener at localhost/36951] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-18 18:55:39,941 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x118e131f to 127.0.0.1:65091 2023-04-18 18:55:39,941 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:55:39,943 INFO [Listener at localhost/36951] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-18 18:55:39,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-18 18:55:39,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-18 18:55:39,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-18 18:55:39,950 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844139950"}]},"ts":"1681844139950"} 2023-04-18 18:55:39,951 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-18 18:55:39,953 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-18 18:55:39,954 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9efbb28428802c9e45d9365ba7ee3aa, UNASSIGN}] 2023-04-18 18:55:39,955 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9efbb28428802c9e45d9365ba7ee3aa, UNASSIGN 2023-04-18 18:55:39,955 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=a9efbb28428802c9e45d9365ba7ee3aa, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:39,955 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844139955"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681844139955"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681844139955"}]},"ts":"1681844139955"} 2023-04-18 18:55:39,956 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952}] 2023-04-18 18:55:40,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-18 18:55:40,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:40,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a9efbb28428802c9e45d9365ba7ee3aa, disabling compactions & flushes 2023-04-18 18:55:40,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:40,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:40,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. after waiting 0 ms 2023-04-18 18:55:40,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:40,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing a9efbb28428802c9e45d9365ba7ee3aa 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-18 18:55:40,109 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=A 2023-04-18 18:55:40,109 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:40,109 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=B 2023-04-18 18:55:40,109 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:40,109 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9efbb28428802c9e45d9365ba7ee3aa, store=C 2023-04-18 18:55:40,109 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:40,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304184227017983e84de28523662d44cbd687 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304184227017983e84de28523662d44cbd687 2023-04-18 18:55:40,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-18 18:55:40,341 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-18 18:55:40,367 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/f074f894c6a643d0841190813863f2ae, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/199c75b353ba4aeaa5100720ca0ae64d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/7b7bac1a14dc43c993bcb5a6a77eb633, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/d44a03233f8142e58aafa3eabf4fb118, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/a99ebd316403454d92eb39a6265cf586, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/1942ed4fc7974b8ea96011a1c2bd1ad4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/9e6076e8f60d45b4bc2941e25802d0db, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/0b67a3bc8dee4444bef4cd1c1f42d191, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/12e6f0cdb7024c37bdc91a38e59f749e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/183ad8c410334613905bd45f0d43fc97, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/36a4c7816ca84ac8bf0be74bbdc1f7e9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/69fa5a0081274eb9896a91d2430fa235, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/7c549d28f6e84c648ab26ffd57113843, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/f38fe752e6cd43cf9fa66dbfb6a6cb1e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/ae6f6e278c0c482ead6df0f76da957e3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/3cbf946484294b6887d3e12cb2f9f053, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/8f5e549b31274355b3d3b32aa8479191, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/817d1a6a1bc1410eb0b8e42caf8ccf1e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/46bb4cac20124a6286de218c03d445af, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/bd86421651964963a5a30d8d98dfa3fc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/64a12d986c3e4b8182d269c771b9bdce, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/65533d45e4a146a4a960926fe4a5faea, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/06758fbed5e84335852e9d345ae5df1f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/c4c4ecaac5dd4ccd9abc2d7a5c13f6ef, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/bfc743b7bc1d46668e64efa179b06a31, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/0daad943a3ff47e2b6e3c5fd701d5377, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/31cfdfb385a443fbb6f30b16443c4758] to archive 2023-04-18 18:55:40,368 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 18:55:40,370 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/f074f894c6a643d0841190813863f2ae to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/f074f894c6a643d0841190813863f2ae 2023-04-18 18:55:40,371 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/199c75b353ba4aeaa5100720ca0ae64d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/199c75b353ba4aeaa5100720ca0ae64d 2023-04-18 18:55:40,372 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/7b7bac1a14dc43c993bcb5a6a77eb633 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/7b7bac1a14dc43c993bcb5a6a77eb633 2023-04-18 18:55:40,373 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/d44a03233f8142e58aafa3eabf4fb118 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/d44a03233f8142e58aafa3eabf4fb118 2023-04-18 18:55:40,373 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/a99ebd316403454d92eb39a6265cf586 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/a99ebd316403454d92eb39a6265cf586 2023-04-18 18:55:40,374 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/1942ed4fc7974b8ea96011a1c2bd1ad4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/1942ed4fc7974b8ea96011a1c2bd1ad4 2023-04-18 18:55:40,375 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/9e6076e8f60d45b4bc2941e25802d0db to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/9e6076e8f60d45b4bc2941e25802d0db 2023-04-18 18:55:40,376 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/0b67a3bc8dee4444bef4cd1c1f42d191 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/0b67a3bc8dee4444bef4cd1c1f42d191 2023-04-18 18:55:40,377 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/12e6f0cdb7024c37bdc91a38e59f749e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/12e6f0cdb7024c37bdc91a38e59f749e 2023-04-18 18:55:40,378 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/183ad8c410334613905bd45f0d43fc97 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/183ad8c410334613905bd45f0d43fc97 2023-04-18 18:55:40,378 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/36a4c7816ca84ac8bf0be74bbdc1f7e9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/36a4c7816ca84ac8bf0be74bbdc1f7e9 2023-04-18 18:55:40,379 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/69fa5a0081274eb9896a91d2430fa235 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/69fa5a0081274eb9896a91d2430fa235 2023-04-18 18:55:40,380 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/7c549d28f6e84c648ab26ffd57113843 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/7c549d28f6e84c648ab26ffd57113843 2023-04-18 18:55:40,381 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/f38fe752e6cd43cf9fa66dbfb6a6cb1e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/f38fe752e6cd43cf9fa66dbfb6a6cb1e 2023-04-18 18:55:40,382 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/ae6f6e278c0c482ead6df0f76da957e3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/ae6f6e278c0c482ead6df0f76da957e3 2023-04-18 18:55:40,382 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/3cbf946484294b6887d3e12cb2f9f053 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/3cbf946484294b6887d3e12cb2f9f053 2023-04-18 18:55:40,383 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/8f5e549b31274355b3d3b32aa8479191 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/8f5e549b31274355b3d3b32aa8479191 2023-04-18 18:55:40,384 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/817d1a6a1bc1410eb0b8e42caf8ccf1e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/817d1a6a1bc1410eb0b8e42caf8ccf1e 2023-04-18 18:55:40,385 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/46bb4cac20124a6286de218c03d445af to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/46bb4cac20124a6286de218c03d445af 2023-04-18 18:55:40,385 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/bd86421651964963a5a30d8d98dfa3fc to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/bd86421651964963a5a30d8d98dfa3fc 2023-04-18 18:55:40,386 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/64a12d986c3e4b8182d269c771b9bdce to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/64a12d986c3e4b8182d269c771b9bdce 2023-04-18 18:55:40,387 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/65533d45e4a146a4a960926fe4a5faea to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/65533d45e4a146a4a960926fe4a5faea 2023-04-18 18:55:40,388 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/06758fbed5e84335852e9d345ae5df1f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/06758fbed5e84335852e9d345ae5df1f 2023-04-18 18:55:40,389 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/c4c4ecaac5dd4ccd9abc2d7a5c13f6ef to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/c4c4ecaac5dd4ccd9abc2d7a5c13f6ef 2023-04-18 18:55:40,390 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/bfc743b7bc1d46668e64efa179b06a31 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/bfc743b7bc1d46668e64efa179b06a31 2023-04-18 18:55:40,390 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/0daad943a3ff47e2b6e3c5fd701d5377 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/0daad943a3ff47e2b6e3c5fd701d5377 2023-04-18 18:55:40,391 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/31cfdfb385a443fbb6f30b16443c4758 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/31cfdfb385a443fbb6f30b16443c4758 2023-04-18 18:55:40,396 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/1a05d716fba546d6b402df3a9f5d3a22, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/af8260e1210741789f2cb61c4b00febe, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/d9f398785f1c4dc5a25184f200c4e546, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/c9ff752f5eb34ea78250f286b103a5e4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/520d342c4e61461a8386f1e364a40889, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/df1e6b10289d4625b67aab030ecd245a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/3b8aac906be345ca94072e7cceaf3075, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/41e648254a694b8c93abd52c98dc1330, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/1215107fcb7949c1a54a96c4d110d91a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4e25e21f6d6f4e4abc5dc53a25dd97ce, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4a2d065955f746d08997592d9cefb917, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/d1589ef0e83a4656817efe53f8446e78, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/2fa99029e8fd4908972d97d50edb3e9c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/9cd761978e3a466bb092651821edc497, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/8cfeb418ddad4bf0a28145e66c065c2f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/ca5667c3c8dd4775a7d9155d6113cd07, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/af2aa740d72c4c86b386af6c47211121, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/fd4d20955a254160b5db2b9d78a3f1e4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/ec9b64d68983444aa0e8efef271820cb, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/89664d68a1e0468a8bbfe99b04c72974, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/dd534cd76cb74611b5161d8834994fd2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/484658c8f43240dcb363ee95f43d576c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4d664e6b3fa14c189c2d62ad5e7cd313, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/bed377a52de04854be853181939d44c6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/fefe96a11cd14043b250fb2a16202b3d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4252f9d772744adf937a156f395734a4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/bfdc915a3ee94894805bde249af6c7e7] to archive 2023-04-18 18:55:40,397 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 18:55:40,398 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/1a05d716fba546d6b402df3a9f5d3a22 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/1a05d716fba546d6b402df3a9f5d3a22 2023-04-18 18:55:40,399 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/af8260e1210741789f2cb61c4b00febe to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/af8260e1210741789f2cb61c4b00febe 2023-04-18 18:55:40,399 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/d9f398785f1c4dc5a25184f200c4e546 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/d9f398785f1c4dc5a25184f200c4e546 2023-04-18 18:55:40,400 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/c9ff752f5eb34ea78250f286b103a5e4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/c9ff752f5eb34ea78250f286b103a5e4 2023-04-18 18:55:40,401 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/520d342c4e61461a8386f1e364a40889 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/520d342c4e61461a8386f1e364a40889 2023-04-18 18:55:40,402 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/df1e6b10289d4625b67aab030ecd245a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/df1e6b10289d4625b67aab030ecd245a 2023-04-18 18:55:40,403 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/3b8aac906be345ca94072e7cceaf3075 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/3b8aac906be345ca94072e7cceaf3075 2023-04-18 18:55:40,403 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/41e648254a694b8c93abd52c98dc1330 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/41e648254a694b8c93abd52c98dc1330 2023-04-18 18:55:40,404 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/1215107fcb7949c1a54a96c4d110d91a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/1215107fcb7949c1a54a96c4d110d91a 2023-04-18 18:55:40,405 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4e25e21f6d6f4e4abc5dc53a25dd97ce to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4e25e21f6d6f4e4abc5dc53a25dd97ce 2023-04-18 18:55:40,406 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4a2d065955f746d08997592d9cefb917 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4a2d065955f746d08997592d9cefb917 2023-04-18 18:55:40,406 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/d1589ef0e83a4656817efe53f8446e78 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/d1589ef0e83a4656817efe53f8446e78 2023-04-18 18:55:40,407 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/2fa99029e8fd4908972d97d50edb3e9c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/2fa99029e8fd4908972d97d50edb3e9c 2023-04-18 18:55:40,408 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/9cd761978e3a466bb092651821edc497 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/9cd761978e3a466bb092651821edc497 2023-04-18 18:55:40,409 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/8cfeb418ddad4bf0a28145e66c065c2f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/8cfeb418ddad4bf0a28145e66c065c2f 2023-04-18 18:55:40,409 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/ca5667c3c8dd4775a7d9155d6113cd07 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/ca5667c3c8dd4775a7d9155d6113cd07 2023-04-18 18:55:40,410 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/af2aa740d72c4c86b386af6c47211121 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/af2aa740d72c4c86b386af6c47211121 2023-04-18 18:55:40,411 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/fd4d20955a254160b5db2b9d78a3f1e4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/fd4d20955a254160b5db2b9d78a3f1e4 2023-04-18 18:55:40,412 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/ec9b64d68983444aa0e8efef271820cb to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/ec9b64d68983444aa0e8efef271820cb 2023-04-18 18:55:40,413 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/89664d68a1e0468a8bbfe99b04c72974 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/89664d68a1e0468a8bbfe99b04c72974 2023-04-18 18:55:40,413 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/dd534cd76cb74611b5161d8834994fd2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/dd534cd76cb74611b5161d8834994fd2 2023-04-18 18:55:40,414 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/484658c8f43240dcb363ee95f43d576c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/484658c8f43240dcb363ee95f43d576c 2023-04-18 18:55:40,415 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4d664e6b3fa14c189c2d62ad5e7cd313 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4d664e6b3fa14c189c2d62ad5e7cd313 2023-04-18 18:55:40,416 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/bed377a52de04854be853181939d44c6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/bed377a52de04854be853181939d44c6 2023-04-18 18:55:40,416 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/fefe96a11cd14043b250fb2a16202b3d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/fefe96a11cd14043b250fb2a16202b3d 2023-04-18 18:55:40,417 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4252f9d772744adf937a156f395734a4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/4252f9d772744adf937a156f395734a4 2023-04-18 18:55:40,418 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/bfdc915a3ee94894805bde249af6c7e7 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/bfdc915a3ee94894805bde249af6c7e7 2023-04-18 18:55:40,422 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/ccd6d5bd04a341f683195239a4de2a5a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/80cd6fe4f0c344ad8fef4fe7dc54399b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/7809733d5fa34f918766c9bb396eac15, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/163c95d502b74223865c5446ca2ad0bd, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/fb0212c1c6164d159968c4fbff32aff9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/fbe3d2016d2a49e0890fce60229c58f4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/8f54fbb7a1434298b47c16ad436c8513, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/861752b0bf214639a15f3acda0435bef, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/0a8716fbb9d1414c9ffff9846c0543ba, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/179c96d7294346b88625dc03b589b474, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/254e0118c0cb40e2ac443b50b89c673d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/92d76af2363c4b83a3d0d29566720bc6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/a144da35e7944650a008e19e2cf68896, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/bec75a94a61a466db9a9f8d062a6afbc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/5eb57c6e30a14f0e9de5d80fd1b32913, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/c9314107e01b4661920e7dfd29ec566f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/ea37117b79db411e99ba0db84e79b9f9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/4cfa3c8d328f4673b1e404c9f46797b8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/e1083056419c418fbc2d295519fd10f4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/d7e9a69548974c528809147008071756, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/2cac2a66610446f6b87faf31e1ab1f1e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/e0a8b19ca87344f18bade7f28a7cf8c6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/477c5624ebc0488ca452ca77c70a6daa, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/e2568054617b4c2e814f0daf18ff5b85, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/efcc8bdba4454419b3f73c548693df0d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/6d15867aa3044893a8d61a191237d5d1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/45db1eb210824b9283af200075ed39d3] to archive 2023-04-18 18:55:40,423 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 18:55:40,424 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/ccd6d5bd04a341f683195239a4de2a5a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/ccd6d5bd04a341f683195239a4de2a5a 2023-04-18 18:55:40,425 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/80cd6fe4f0c344ad8fef4fe7dc54399b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/80cd6fe4f0c344ad8fef4fe7dc54399b 2023-04-18 18:55:40,426 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/7809733d5fa34f918766c9bb396eac15 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/7809733d5fa34f918766c9bb396eac15 2023-04-18 18:55:40,427 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/163c95d502b74223865c5446ca2ad0bd to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/163c95d502b74223865c5446ca2ad0bd 2023-04-18 18:55:40,427 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/fb0212c1c6164d159968c4fbff32aff9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/fb0212c1c6164d159968c4fbff32aff9 2023-04-18 18:55:40,428 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/fbe3d2016d2a49e0890fce60229c58f4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/fbe3d2016d2a49e0890fce60229c58f4 2023-04-18 18:55:40,429 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/8f54fbb7a1434298b47c16ad436c8513 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/8f54fbb7a1434298b47c16ad436c8513 2023-04-18 18:55:40,430 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/861752b0bf214639a15f3acda0435bef to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/861752b0bf214639a15f3acda0435bef 2023-04-18 18:55:40,431 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/0a8716fbb9d1414c9ffff9846c0543ba to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/0a8716fbb9d1414c9ffff9846c0543ba 2023-04-18 18:55:40,432 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/179c96d7294346b88625dc03b589b474 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/179c96d7294346b88625dc03b589b474 2023-04-18 18:55:40,433 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/254e0118c0cb40e2ac443b50b89c673d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/254e0118c0cb40e2ac443b50b89c673d 2023-04-18 18:55:40,434 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/92d76af2363c4b83a3d0d29566720bc6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/92d76af2363c4b83a3d0d29566720bc6 2023-04-18 18:55:40,435 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/a144da35e7944650a008e19e2cf68896 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/a144da35e7944650a008e19e2cf68896 2023-04-18 18:55:40,436 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/bec75a94a61a466db9a9f8d062a6afbc to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/bec75a94a61a466db9a9f8d062a6afbc 2023-04-18 18:55:40,436 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/5eb57c6e30a14f0e9de5d80fd1b32913 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/5eb57c6e30a14f0e9de5d80fd1b32913 2023-04-18 18:55:40,437 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/c9314107e01b4661920e7dfd29ec566f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/c9314107e01b4661920e7dfd29ec566f 2023-04-18 18:55:40,438 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/ea37117b79db411e99ba0db84e79b9f9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/ea37117b79db411e99ba0db84e79b9f9 2023-04-18 18:55:40,439 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/4cfa3c8d328f4673b1e404c9f46797b8 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/4cfa3c8d328f4673b1e404c9f46797b8 2023-04-18 18:55:40,440 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/e1083056419c418fbc2d295519fd10f4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/e1083056419c418fbc2d295519fd10f4 2023-04-18 18:55:40,440 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/d7e9a69548974c528809147008071756 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/d7e9a69548974c528809147008071756 2023-04-18 18:55:40,441 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/2cac2a66610446f6b87faf31e1ab1f1e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/2cac2a66610446f6b87faf31e1ab1f1e 2023-04-18 18:55:40,442 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/e0a8b19ca87344f18bade7f28a7cf8c6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/e0a8b19ca87344f18bade7f28a7cf8c6 2023-04-18 18:55:40,443 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/477c5624ebc0488ca452ca77c70a6daa to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/477c5624ebc0488ca452ca77c70a6daa 2023-04-18 18:55:40,444 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/e2568054617b4c2e814f0daf18ff5b85 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/e2568054617b4c2e814f0daf18ff5b85 2023-04-18 18:55:40,444 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/efcc8bdba4454419b3f73c548693df0d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/efcc8bdba4454419b3f73c548693df0d 2023-04-18 18:55:40,445 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/6d15867aa3044893a8d61a191237d5d1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/6d15867aa3044893a8d61a191237d5d1 2023-04-18 18:55:40,446 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/45db1eb210824b9283af200075ed39d3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/45db1eb210824b9283af200075ed39d3 2023-04-18 18:55:40,539 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=573, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/d103f44c42cb4b478e1f0040263e2464 2023-04-18 18:55:40,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-18 18:55:40,949 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/c309ebab15c64be596c652baacedfb55 2023-04-18 18:55:41,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-18 18:55:41,359 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/2186ec49fd3f45308a10051a4ab44986 2023-04-18 18:55:41,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/A/d103f44c42cb4b478e1f0040263e2464 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/d103f44c42cb4b478e1f0040263e2464 2023-04-18 18:55:41,366 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/d103f44c42cb4b478e1f0040263e2464, entries=100, sequenceid=573, filesize=18.6 K 2023-04-18 18:55:41,366 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/B/c309ebab15c64be596c652baacedfb55 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/c309ebab15c64be596c652baacedfb55 2023-04-18 18:55:41,369 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/c309ebab15c64be596c652baacedfb55, entries=100, sequenceid=573, filesize=9.5 K 2023-04-18 18:55:41,369 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/.tmp/C/2186ec49fd3f45308a10051a4ab44986 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/2186ec49fd3f45308a10051a4ab44986 2023-04-18 18:55:41,372 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/2186ec49fd3f45308a10051a4ab44986, entries=100, sequenceid=573, filesize=9.5 K 2023-04-18 18:55:41,372 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for a9efbb28428802c9e45d9365ba7ee3aa in 1264ms, sequenceid=573, compaction requested=false 2023-04-18 18:55:41,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/recovered.edits/576.seqid, newMaxSeqId=576, maxSeqId=4 2023-04-18 18:55:41,382 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa. 2023-04-18 18:55:41,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a9efbb28428802c9e45d9365ba7ee3aa: 2023-04-18 18:55:41,383 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:41,384 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=a9efbb28428802c9e45d9365ba7ee3aa, regionState=CLOSED 2023-04-18 18:55:41,384 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844141384"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681844141384"}]},"ts":"1681844141384"} 2023-04-18 18:55:41,386 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-18 18:55:41,386 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure a9efbb28428802c9e45d9365ba7ee3aa, server=jenkins-hbase4.apache.org,34053,1681844018952 in 1.4290 sec 2023-04-18 18:55:41,387 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-18 18:55:41,387 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9efbb28428802c9e45d9365ba7ee3aa, UNASSIGN in 1.4320 sec 2023-04-18 18:55:41,387 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844141387"}]},"ts":"1681844141387"} 2023-04-18 18:55:41,388 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-18 18:55:41,389 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-18 18:55:41,391 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.4440 sec 2023-04-18 18:55:42,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-18 18:55:42,054 INFO [Listener at localhost/36951] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-18 18:55:42,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-18 18:55:42,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:55:42,057 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:55:42,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-18 18:55:42,057 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:55:42,061 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:42,062 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A, FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B, FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C, FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/recovered.edits] 2023-04-18 18:55:42,064 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/516541e819bb43e0b865097342055b37 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/516541e819bb43e0b865097342055b37 2023-04-18 18:55:42,065 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/d103f44c42cb4b478e1f0040263e2464 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/A/d103f44c42cb4b478e1f0040263e2464 2023-04-18 18:55:42,068 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/74380f3a4d97444f9909c9e75be3e14a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/74380f3a4d97444f9909c9e75be3e14a 2023-04-18 18:55:42,069 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/c309ebab15c64be596c652baacedfb55 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/B/c309ebab15c64be596c652baacedfb55 2023-04-18 18:55:42,070 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/0c17e26348e2441993714f0aed5a3196 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/0c17e26348e2441993714f0aed5a3196 2023-04-18 18:55:42,071 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/2186ec49fd3f45308a10051a4ab44986 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/C/2186ec49fd3f45308a10051a4ab44986 2023-04-18 18:55:42,073 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/recovered.edits/576.seqid to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa/recovered.edits/576.seqid 2023-04-18 18:55:42,074 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/a9efbb28428802c9e45d9365ba7ee3aa 2023-04-18 18:55:42,074 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 18:55:42,074 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-18 18:55:42,075 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-18 18:55:42,077 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041814dae8db102844b68ee2948e9098d11f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041814dae8db102844b68ee2948e9098d11f 2023-04-18 18:55:42,078 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418210897ead31849ecb931c011f4e0b76a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418210897ead31849ecb931c011f4e0b76a 2023-04-18 18:55:42,079 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041830055fddff4441038ac2bcce2dbf8a58 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041830055fddff4441038ac2bcce2dbf8a58 2023-04-18 18:55:42,079 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304184227017983e84de28523662d44cbd687 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304184227017983e84de28523662d44cbd687 2023-04-18 18:55:42,080 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304184859e93a62594f94ad38c89952445d74 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304184859e93a62594f94ad38c89952445d74 2023-04-18 18:55:42,081 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041851aacc91f62b4957966c5de7de7992a0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041851aacc91f62b4957966c5de7de7992a0 2023-04-18 18:55:42,082 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418583e36ad7d674a2eae40cb9976d67e6f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418583e36ad7d674a2eae40cb9976d67e6f 2023-04-18 18:55:42,083 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041864a2c80975ca4e3d8721ebb61c2c245d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041864a2c80975ca4e3d8721ebb61c2c245d 2023-04-18 18:55:42,083 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041871f4d47c07cc4177960fdccc9a2fe8a9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041871f4d47c07cc4177960fdccc9a2fe8a9 2023-04-18 18:55:42,084 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304187d8b14af533742679fffda1d576acc64 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304187d8b14af533742679fffda1d576acc64 2023-04-18 18:55:42,085 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041880ae103d51e54faca86014279bcc6f78 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041880ae103d51e54faca86014279bcc6f78 2023-04-18 18:55:42,086 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304188837bcb52a994edb919144f20cfec895 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304188837bcb52a994edb919144f20cfec895 2023-04-18 18:55:42,086 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a001e25f02fe40fd8691c32345871ef6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a001e25f02fe40fd8691c32345871ef6 2023-04-18 18:55:42,087 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a3485983dcdb4d59b1298e792792832a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a3485983dcdb4d59b1298e792792832a 2023-04-18 18:55:42,088 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418ad282f41950b4879b4fb0a4c50870ba6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418ad282f41950b4879b4fb0a4c50870ba6 2023-04-18 18:55:42,088 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418ad6d71c3a7434171ad628227b80e1a3b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418ad6d71c3a7434171ad628227b80e1a3b 2023-04-18 18:55:42,089 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418b50d7c9e326c4a4f86fb2884d67b3a16 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418b50d7c9e326c4a4f86fb2884d67b3a16 2023-04-18 18:55:42,090 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418ccdbf5aaf6b742ea87956207266c551e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418ccdbf5aaf6b742ea87956207266c551e 2023-04-18 18:55:42,091 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418df9f2ec7d24e4fcaabdba7ee99191afb to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418df9f2ec7d24e4fcaabdba7ee99191afb 2023-04-18 18:55:42,091 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e3078a5ea06043a6b9e3ed377f8325d0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e3078a5ea06043a6b9e3ed377f8325d0 2023-04-18 18:55:42,092 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418eca83accecec4acda86621899c5dea12 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418eca83accecec4acda86621899c5dea12 2023-04-18 18:55:42,093 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418fd6da59d0fb64850b5c09359526bb86f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418fd6da59d0fb64850b5c09359526bb86f 2023-04-18 18:55:42,093 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-18 18:55:42,095 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:55:42,096 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-18 18:55:42,098 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-18 18:55:42,099 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:55:42,099 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-18 18:55:42,099 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681844142099"}]},"ts":"9223372036854775807"} 2023-04-18 18:55:42,100 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-18 18:55:42,100 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a9efbb28428802c9e45d9365ba7ee3aa, NAME => 'TestAcidGuarantees,,1681844106352.a9efbb28428802c9e45d9365ba7ee3aa.', STARTKEY => '', ENDKEY => ''}] 2023-04-18 18:55:42,100 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-18 18:55:42,100 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681844142100"}]},"ts":"9223372036854775807"} 2023-04-18 18:55:42,101 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-18 18:55:42,103 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:55:42,104 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 49 msec 2023-04-18 18:55:42,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-18 18:55:42,158 INFO [Listener at localhost/36951] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-18 18:55:42,170 INFO [Listener at localhost/36951] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=229 (was 227) - Thread LEAK? -, OpenFileDescriptor=517 (was 565), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=320 (was 375), ProcessCount=172 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=6426 (was 6586) 2023-04-18 18:55:42,181 INFO [Listener at localhost/36951] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=229, OpenFileDescriptor=517, MaxFileDescriptor=60000, SystemLoadAverage=320, ProcessCount=172, AvailableMemoryMB=6427 2023-04-18 18:55:42,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-18 18:55:42,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-18 18:55:42,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-18 18:55:42,186 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-18 18:55:42,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-18 18:55:42,186 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-18 18:55:42,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-18 18:55:42,187 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:42,188 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66 empty. 2023-04-18 18:55:42,188 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:42,188 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 18:55:42,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-18 18:55:42,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-18 18:55:42,595 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-18 18:55:42,596 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 814b3c470b4969ca1c95de968f5dec66, NAME => 'TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp 2023-04-18 18:55:42,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-18 18:55:43,006 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 18:55:43,006 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 814b3c470b4969ca1c95de968f5dec66, disabling compactions & flushes 2023-04-18 18:55:43,006 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:43,006 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:43,006 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. after waiting 0 ms 2023-04-18 18:55:43,006 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:43,006 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:43,006 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:43,008 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-18 18:55:43,009 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844143009"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681844143009"}]},"ts":"1681844143009"} 2023-04-18 18:55:43,010 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-18 18:55:43,011 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-18 18:55:43,011 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844143011"}]},"ts":"1681844143011"} 2023-04-18 18:55:43,012 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-18 18:55:43,015 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=814b3c470b4969ca1c95de968f5dec66, ASSIGN}] 2023-04-18 18:55:43,017 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=814b3c470b4969ca1c95de968f5dec66, ASSIGN 2023-04-18 18:55:43,018 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=814b3c470b4969ca1c95de968f5dec66, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34053,1681844018952; forceNewPlan=false, retain=false 2023-04-18 18:55:43,169 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=814b3c470b4969ca1c95de968f5dec66, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:43,169 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844143169"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681844143169"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681844143169"}]},"ts":"1681844143169"} 2023-04-18 18:55:43,170 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952}] 2023-04-18 18:55:43,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-18 18:55:43,324 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:43,324 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 814b3c470b4969ca1c95de968f5dec66, NAME => 'TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.', STARTKEY => '', ENDKEY => ''} 2023-04-18 18:55:43,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:43,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 18:55:43,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:43,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:43,326 INFO [StoreOpener-814b3c470b4969ca1c95de968f5dec66-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:43,327 DEBUG [StoreOpener-814b3c470b4969ca1c95de968f5dec66-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A 2023-04-18 18:55:43,327 DEBUG [StoreOpener-814b3c470b4969ca1c95de968f5dec66-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A 2023-04-18 18:55:43,327 INFO [StoreOpener-814b3c470b4969ca1c95de968f5dec66-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:55:43,327 INFO [StoreOpener-814b3c470b4969ca1c95de968f5dec66-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 814b3c470b4969ca1c95de968f5dec66 columnFamilyName A 2023-04-18 18:55:43,328 INFO [StoreOpener-814b3c470b4969ca1c95de968f5dec66-1] regionserver.HStore(310): Store=814b3c470b4969ca1c95de968f5dec66/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:55:43,328 INFO [StoreOpener-814b3c470b4969ca1c95de968f5dec66-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:43,329 DEBUG [StoreOpener-814b3c470b4969ca1c95de968f5dec66-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B 2023-04-18 18:55:43,329 DEBUG [StoreOpener-814b3c470b4969ca1c95de968f5dec66-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B 2023-04-18 18:55:43,330 INFO [StoreOpener-814b3c470b4969ca1c95de968f5dec66-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:55:43,330 INFO [StoreOpener-814b3c470b4969ca1c95de968f5dec66-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 814b3c470b4969ca1c95de968f5dec66 columnFamilyName B 2023-04-18 18:55:43,330 INFO [StoreOpener-814b3c470b4969ca1c95de968f5dec66-1] regionserver.HStore(310): Store=814b3c470b4969ca1c95de968f5dec66/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:55:43,330 INFO [StoreOpener-814b3c470b4969ca1c95de968f5dec66-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:43,331 DEBUG [StoreOpener-814b3c470b4969ca1c95de968f5dec66-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C 2023-04-18 18:55:43,332 DEBUG [StoreOpener-814b3c470b4969ca1c95de968f5dec66-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C 2023-04-18 18:55:43,332 INFO [StoreOpener-814b3c470b4969ca1c95de968f5dec66-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:55:43,332 INFO [StoreOpener-814b3c470b4969ca1c95de968f5dec66-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 814b3c470b4969ca1c95de968f5dec66 columnFamilyName C 2023-04-18 18:55:43,332 INFO [StoreOpener-814b3c470b4969ca1c95de968f5dec66-1] regionserver.HStore(310): Store=814b3c470b4969ca1c95de968f5dec66/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:55:43,332 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:43,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:43,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:43,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 18:55:43,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:43,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-18 18:55:43,338 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 814b3c470b4969ca1c95de968f5dec66; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59351446, jitterRate=-0.1155945360660553}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 18:55:43,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:43,338 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., pid=49, masterSystemTime=1681844143322 2023-04-18 18:55:43,339 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:43,339 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:43,340 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=814b3c470b4969ca1c95de968f5dec66, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:43,340 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844143340"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681844143340"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681844143340"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681844143340"}]},"ts":"1681844143340"} 2023-04-18 18:55:43,342 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-18 18:55:43,342 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 in 171 msec 2023-04-18 18:55:43,343 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-18 18:55:43,343 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=814b3c470b4969ca1c95de968f5dec66, ASSIGN in 327 msec 2023-04-18 18:55:43,343 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-18 18:55:43,344 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844143343"}]},"ts":"1681844143343"} 2023-04-18 18:55:43,344 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-18 18:55:43,346 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-18 18:55:43,347 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1620 sec 2023-04-18 18:55:44,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-18 18:55:44,292 INFO [Listener at localhost/36951] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-18 18:55:44,292 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x3fb41c50 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:44,301 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e0b6187, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:44,302 DEBUG [Listener at localhost/36951] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:44,305 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60762, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:44,306 DEBUG [Listener at localhost/36951] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-18 18:55:44,308 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36496, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-18 18:55:44,309 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x357cb237 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:44,314 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ab56e54, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:44,315 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x4539ed0c to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:44,327 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15840296, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:44,328 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x32cd672d to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:44,338 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3816bdf5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:44,339 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x09b8ff30 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:44,348 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50150525, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:44,348 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x4c050db8 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:44,354 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28a3a77, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:44,355 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x4cefb012 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:44,360 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f65627a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:44,361 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x24a917df to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:44,369 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15f74c78, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:44,370 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x78fa3dee to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:44,376 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1401cac1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:44,377 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x36d2d826 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:44,383 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f6e865f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:44,384 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x6871d7d8 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:55:44,390 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@48c22118, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:55:44,405 DEBUG [hconnection-0xced259e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:44,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:44,428 DEBUG [hconnection-0x329cdcbd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:44,429 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60788, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:44,432 DEBUG [hconnection-0x3218b3a6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:44,437 DEBUG [hconnection-0x3724a60-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:44,437 DEBUG [hconnection-0x316555aa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:44,438 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60806, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:44,440 DEBUG [hconnection-0x149988fd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:44,443 DEBUG [hconnection-0x5cb5fabb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:44,447 DEBUG [hconnection-0x75017dc2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:44,447 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60778, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:44,447 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60794, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:44,448 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60796, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:44,448 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60810, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:44,449 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60820, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:44,452 DEBUG [hconnection-0x1a51d3c9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:44,452 DEBUG [hconnection-0x198ace3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:55:44,457 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60816, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:44,458 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60844, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:44,459 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60834, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:55:44,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:44,469 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-18 18:55:44,469 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:44,469 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:44,469 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:44,469 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:44,470 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:44,470 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:44,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:44,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:44,470 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:44,470 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:44,474 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:44,474 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:44,477 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,478 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:44,479 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:44,479 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:44,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:44,479 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,479 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:44,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:44,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:44,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:44,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:44,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:44,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:44,496 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:44,496 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:44,496 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:44,501 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:44,501 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:44,501 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:44,501 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:44,501 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:44,501 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:44,501 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:44,501 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:44,501 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1681844144482, max timestamp=1681844144503Num uniques 859; 2023-04-18 18:55:44,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 18:55:44,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1681844144482, max timestamp=1681844144503Num uniques 859; 2023-04-18 18:55:44,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 18:55:44,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1681844144482, max timestamp=1681844144503Num uniques 859; 2023-04-18 18:55:44,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 18:55:44,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:44,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844204513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:44,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844204514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:44,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844204515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:44,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:44,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844204515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844204516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,528 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:44,528 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:44,530 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,530 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,530 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,530 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:44,531 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/461ebbc7e7754e7b906713bef02403b5 2023-04-18 18:55:44,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:44,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:44,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:44,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:44,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@393ebd82[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:44,538 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:44,538 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,539 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,539 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:44,539 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:44,539 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:44,539 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,539 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:44,542 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:44,542 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:44,542 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:44,542 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:44,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:44,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:44,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:44,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:44,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:44,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:44,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:44,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:44,544 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:44,544 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:44,544 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:44,544 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:44,544 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:44,545 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,545 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,546 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:44,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:44,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:44,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,546 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:44,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:44,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:44,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:44,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:44,547 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:44,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:44,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,549 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,549 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:44,549 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:44,549 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,549 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,549 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:44,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:44,549 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:44,549 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:44,549 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,550 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:44,550 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:44,550 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:44,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:44,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:44,577 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/a01df87a0d524a5183b70a24f8609766 2023-04-18 18:55:44,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:44,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844204616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:44,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844204618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:44,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844204618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:44,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844204618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:44,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844204618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,650 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:44,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:44,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:44,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:44,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:44,678 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:44,678 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:44,678 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:44,678 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:44,678 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,678 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:44,684 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:44,684 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:44,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:44,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:44,685 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,685 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:44,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:44,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:44,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:44,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:44,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:44,691 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:44,691 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:44,691 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:44,700 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:44,700 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:44,700 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:44,700 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:44,700 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:44,700 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:44,700 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:44,700 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:44,700 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,704 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:44,704 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:44,709 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,709 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:44,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:44,710 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,710 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:44,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:44,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:44,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c83900f[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:44,711 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:44,711 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,713 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,714 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:44,714 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:44,714 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:44,714 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,714 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:44,715 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,715 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:44,715 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:44,715 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:44,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:44,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:44,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:44,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:44,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:44,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:44,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:44,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:44,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:44,717 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:44,717 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:44,717 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:44,718 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:44,718 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:44,719 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,719 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:44,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,719 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:44,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:44,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:44,720 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:44,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:44,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:44,721 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:44,729 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:44,729 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:44,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:44,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:44,729 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,729 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:44,729 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:44,729 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,729 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,729 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,729 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,729 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:44,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:44,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:44,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:44,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:44,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:44,730 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:44,730 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:44,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:44,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844204821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:44,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844204823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:44,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844204823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:44,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844204823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:44,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844204824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,830 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:44,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:44,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:44,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:44,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:44,848 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:44,848 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:44,848 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:44,848 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:44,852 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,852 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:44,854 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,854 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:44,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:44,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:44,854 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,854 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:44,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:44,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:44,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:44,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:44,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:44,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:44,856 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:44,856 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:44,856 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:44,860 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:44,860 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:44,860 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:44,860 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,863 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:44,863 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:44,865 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,865 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:44,865 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,865 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:44,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:44,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:44,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dd336fb[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:44,870 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:44,870 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,871 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:44,871 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:44,871 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:44,871 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,871 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:44,873 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:44,874 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:44,874 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:44,874 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:44,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:44,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:44,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:44,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:44,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:44,878 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:44,878 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:44,878 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:44,878 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:44,879 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:44,881 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,881 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:44,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,881 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:44,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:44,881 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,882 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,884 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:44,884 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:44,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:44,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:44,884 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:44,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:44,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:44,884 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:44,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:44,884 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:44,888 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,888 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:44,888 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:44,888 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:44,888 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,888 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:44,888 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:44,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:44,985 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:44,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:44,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:45,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:45,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:45,002 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:45,002 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:45,002 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:45,002 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:45,003 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,003 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,005 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,005 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:45,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:45,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:45,006 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,006 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:45,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:45,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:45,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:45,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:45,007 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:45,007 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:45,007 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:45,013 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:45,013 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:45,014 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:45,014 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:45,014 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:45,014 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:45,014 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:45,014 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:45,014 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,014 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:45,014 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:45,017 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,017 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:45,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:45,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dc668f8[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:45,019 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:45,019 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,020 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:45,021 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:45,021 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,021 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:45,021 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:45,022 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:45,022 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:45,024 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:45,024 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:45,024 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/3542937df8644932baf310cf457aab84 2023-04-18 18:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:45,030 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/461ebbc7e7754e7b906713bef02403b5 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/461ebbc7e7754e7b906713bef02403b5 2023-04-18 18:55:45,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:45,030 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:45,030 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:45,031 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:45,031 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:45,031 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:45,032 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,032 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:45,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,032 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:45,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:45,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:45,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:45,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:45,036 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,040 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,045 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/461ebbc7e7754e7b906713bef02403b5, entries=150, sequenceid=13, filesize=11.6 K 2023-04-18 18:55:45,045 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:45,045 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:45,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:45,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:45,045 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,045 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:45,045 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:45,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:45,046 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,046 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,046 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,046 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,046 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,046 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:45,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:45,046 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:45,046 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/a01df87a0d524a5183b70a24f8609766 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/a01df87a0d524a5183b70a24f8609766 2023-04-18 18:55:45,050 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/a01df87a0d524a5183b70a24f8609766, entries=150, sequenceid=13, filesize=11.6 K 2023-04-18 18:55:45,050 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/3542937df8644932baf310cf457aab84 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/3542937df8644932baf310cf457aab84 2023-04-18 18:55:45,063 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/3542937df8644932baf310cf457aab84, entries=150, sequenceid=13, filesize=11.6 K 2023-04-18 18:55:45,064 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=134.18 KB/137400 for 814b3c470b4969ca1c95de968f5dec66 in 595ms, sequenceid=13, compaction requested=false 2023-04-18 18:55:45,064 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:45,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:45,131 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-18 18:55:45,132 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:45,132 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:45,132 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:45,132 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:45,132 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:45,132 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:45,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844205136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,146 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:45,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:45,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844205137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844205140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844205139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844205143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:45,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:45,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:45,164 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:45,164 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:45,164 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:45,164 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:45,165 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,165 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,166 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:45,166 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,167 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,167 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:45,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:45,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:45,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:45,168 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:45,168 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:45,168 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:45,174 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:45,174 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:45,176 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:45,176 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:45,176 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:45,176 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:45,176 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:45,176 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:45,176 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,176 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:45,176 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:45,178 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:45,179 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:45,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:45,179 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,179 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:45,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:45,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:45,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11581c6f[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:45,180 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:45,180 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,181 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,182 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:45,182 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:45,182 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:45,182 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,182 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:45,185 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:45,185 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:45,185 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:45,185 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:45,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:45,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:45,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:45,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:45,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:45,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:45,187 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:45,187 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:45,187 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:45,187 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:45,188 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:45,189 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,189 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:45,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:45,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,189 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:45,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:45,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:45,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:45,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,190 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:45,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,192 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:45,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,200 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:45,200 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:45,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:45,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:45,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:45,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:45,200 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:45,200 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:45,200 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:45,200 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,200 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,200 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,201 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,201 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,201 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,201 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:45,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:45,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844205248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844205248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844205249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844205249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844205250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,300 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:45,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:45,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:45,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:45,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:45,305 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:45,305 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:45,306 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:45,306 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:45,306 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,306 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,308 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,308 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:45,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:45,309 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,309 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:45,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:45,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:45,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:45,319 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:45,319 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:45,319 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:45,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:45,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:45,320 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:45,320 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:45,320 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:45,320 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:45,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:45,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:45,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:45,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:45,322 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,322 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:45,323 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,323 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:45,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:45,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:45,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:45,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:45,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@519bf5a3[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:45,325 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:45,325 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,327 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,327 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,327 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:45,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,327 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:45,327 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:45,327 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:45,329 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,329 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:45,329 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:45,329 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:45,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:45,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:45,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:45,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:45,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:45,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:45,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:45,333 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:45,333 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:45,333 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:45,333 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:45,334 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:45,336 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,336 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:45,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:45,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:45,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,338 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:45,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:45,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:45,339 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:45,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,342 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:45,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,344 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:45,344 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:45,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:45,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:45,345 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,345 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,345 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,345 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:45,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:45,346 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:45,346 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:45,345 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:45,345 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,346 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,346 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:45,446 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:45,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:45,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:45,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:45,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:45,451 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:45,451 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:45,451 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:45,451 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:45,451 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,451 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,454 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:45,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,454 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:45,455 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,455 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:45,457 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:45,457 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:45,457 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:45,457 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:45,458 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:45,458 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:45,458 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:45,458 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:45,458 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:45,458 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:45,458 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:45,458 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:45,458 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:45,458 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,458 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:45,458 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:45,460 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,460 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,460 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:45,461 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:45,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844205456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844205456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844205456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:45,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:45,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45eb6585[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:45,464 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:45,464 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844205460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,466 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,466 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:45,466 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:45,466 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:45,467 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,467 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:45,468 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:45,468 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:45,468 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,468 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:45,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:45,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:45,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844205461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:45,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:45,472 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:45,472 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:45,472 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:45,472 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:45,472 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:45,476 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,476 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:45,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:45,476 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:45,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:45,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:45,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:45,478 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:45,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,480 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:45,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,482 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:45,482 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:45,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:45,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:45,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:45,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:45,483 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,483 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,483 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,483 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,483 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,483 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:45,483 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:45,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:45,483 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,484 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:45,484 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:45,554 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/9b4e1b76613f4469b69c4ea7c3b418fd 2023-04-18 18:55:45,578 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/f2dcce9f481343deba517775f3ddae28 2023-04-18 18:55:45,584 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:45,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:45,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:45,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:45,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:45,596 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:45,596 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:45,596 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:45,596 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:45,597 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,597 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,599 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:45,599 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:45,599 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,599 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:45,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:45,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:45,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:45,620 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:45,620 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:45,620 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:45,628 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:45,628 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:45,628 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:45,628 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:45,628 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:45,628 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:45,628 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:45,628 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:45,628 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,628 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:45,628 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:45,630 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:45,630 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:45,631 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,631 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:45,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:45,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b045468[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:45,633 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:45,633 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,634 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,635 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/7cbb0bd13be54e298ce09994fcd1d2d7 2023-04-18 18:55:45,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,635 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:45,635 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:45,635 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:45,635 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,635 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:45,638 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,638 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:45,638 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:45,638 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:45,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:45,640 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/9b4e1b76613f4469b69c4ea7c3b418fd as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/9b4e1b76613f4469b69c4ea7c3b418fd 2023-04-18 18:55:45,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:45,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:45,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:45,643 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:45,643 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:45,643 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:45,643 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:45,643 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:45,645 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/9b4e1b76613f4469b69c4ea7c3b418fd, entries=250, sequenceid=49, filesize=16.2 K 2023-04-18 18:55:45,648 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/f2dcce9f481343deba517775f3ddae28 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/f2dcce9f481343deba517775f3ddae28 2023-04-18 18:55:45,648 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,648 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:45,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,648 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:45,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,649 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:45,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:45,651 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:45,652 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/f2dcce9f481343deba517775f3ddae28, entries=150, sequenceid=49, filesize=11.6 K 2023-04-18 18:55:45,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,653 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/7cbb0bd13be54e298ce09994fcd1d2d7 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/7cbb0bd13be54e298ce09994fcd1d2d7 2023-04-18 18:55:45,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,655 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:45,655 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:45,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:45,655 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:45,655 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:45,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:45,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:45,655 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:45,655 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:45,655 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,656 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,656 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,656 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,656 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,656 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:45,659 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/7cbb0bd13be54e298ce09994fcd1d2d7, entries=150, sequenceid=49, filesize=11.6 K 2023-04-18 18:55:45,660 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=40.25 KB/41220 for 814b3c470b4969ca1c95de968f5dec66 in 529ms, sequenceid=49, compaction requested=false 2023-04-18 18:55:45,660 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:45,756 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:45,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:45,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:45,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:45,768 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 18:55:45,768 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:45,768 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:45,768 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:45,768 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:45,768 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:45,768 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:45,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:45,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:45,772 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:45,772 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:45,772 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:45,772 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:45,773 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,773 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,777 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:45,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:45,778 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:45,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,778 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,779 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:45,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:45,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:45,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:45,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:45,794 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:45,794 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:45,794 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:45,800 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:45,800 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:45,800 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:45,800 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:45,800 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:45,800 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:45,800 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:45,800 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:45,800 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,808 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:45,808 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:45,810 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,810 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:45,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681844145787, max timestamp=1681844145809Num uniques 859; 2023-04-18 18:55:45,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 18:55:45,811 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:45,811 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:45,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681844145787, max timestamp=1681844145809Num uniques 859; 2023-04-18 18:55:45,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 18:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:45,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:45,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a6e9e03[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:45,813 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:45,813 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844145787, max timestamp=1681844145809Num uniques 859; 2023-04-18 18:55:45,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:55:45,815 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,815 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:45,820 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,820 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:45,820 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:45,820 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:45,822 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,822 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:45,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:45,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:45,822 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:45,822 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:45,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:45,826 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:45,826 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:45,826 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:45,826 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:45,826 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:45,828 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:45,828 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,828 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:45,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:45,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:45,829 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,831 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,834 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/b5f3dd6e801c4bf3944f99f66a186b27 2023-04-18 18:55:45,836 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:45,836 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:45,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:45,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:45,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,838 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:45,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:45,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:45,839 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,839 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,839 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,839 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:45,839 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:45,862 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/d332a09c4dcc4611877909baab6f67e1 2023-04-18 18:55:45,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844205847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844205849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844205862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,875 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/182148ae60634e019470f59bbab0837d 2023-04-18 18:55:45,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844205863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844205864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,879 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/b5f3dd6e801c4bf3944f99f66a186b27 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b5f3dd6e801c4bf3944f99f66a186b27 2023-04-18 18:55:45,882 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b5f3dd6e801c4bf3944f99f66a186b27, entries=200, sequenceid=59, filesize=13.9 K 2023-04-18 18:55:45,883 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/d332a09c4dcc4611877909baab6f67e1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/d332a09c4dcc4611877909baab6f67e1 2023-04-18 18:55:45,886 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/d332a09c4dcc4611877909baab6f67e1, entries=150, sequenceid=59, filesize=11.6 K 2023-04-18 18:55:45,887 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/182148ae60634e019470f59bbab0837d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/182148ae60634e019470f59bbab0837d 2023-04-18 18:55:45,892 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/182148ae60634e019470f59bbab0837d, entries=150, sequenceid=59, filesize=11.6 K 2023-04-18 18:55:45,892 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=143.13 KB/146560 for 814b3c470b4969ca1c95de968f5dec66 in 124ms, sequenceid=59, compaction requested=true 2023-04-18 18:55:45,892 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:45,892 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:45,892 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:45,893 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:45,892 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:45,893 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:45,901 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:45,901 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/B is initiating minor compaction (all files) 2023-04-18 18:55:45,901 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/B in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:45,902 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/a01df87a0d524a5183b70a24f8609766, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/f2dcce9f481343deba517775f3ddae28, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/d332a09c4dcc4611877909baab6f67e1] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=34.8 K 2023-04-18 18:55:45,902 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting a01df87a0d524a5183b70a24f8609766, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681844144459 2023-04-18 18:55:45,902 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42678 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:45,902 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting f2dcce9f481343deba517775f3ddae28, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681844144514 2023-04-18 18:55:45,902 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/A is initiating minor compaction (all files) 2023-04-18 18:55:45,902 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/A in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:45,902 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/461ebbc7e7754e7b906713bef02403b5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/9b4e1b76613f4469b69c4ea7c3b418fd, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b5f3dd6e801c4bf3944f99f66a186b27] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=41.7 K 2023-04-18 18:55:45,903 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting d332a09c4dcc4611877909baab6f67e1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681844145136 2023-04-18 18:55:45,903 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 461ebbc7e7754e7b906713bef02403b5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681844144459 2023-04-18 18:55:45,903 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 9b4e1b76613f4469b69c4ea7c3b418fd, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681844144512 2023-04-18 18:55:45,904 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting b5f3dd6e801c4bf3944f99f66a186b27, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681844145133 2023-04-18 18:55:45,917 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#B#compaction#395 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:45,917 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#A#compaction#396 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:45,932 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/8f08bb6524804e65a439c2266cf5e937 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8f08bb6524804e65a439c2266cf5e937 2023-04-18 18:55:45,938 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/B of 814b3c470b4969ca1c95de968f5dec66 into 8f08bb6524804e65a439c2266cf5e937(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:45,938 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:45,938 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/B, priority=13, startTime=1681844145892; duration=0sec 2023-04-18 18:55:45,938 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:45,939 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:45,939 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:45,940 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:45,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:45,940 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/C is initiating minor compaction (all files) 2023-04-18 18:55:45,940 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/C in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:45,940 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/3542937df8644932baf310cf457aab84, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/7cbb0bd13be54e298ce09994fcd1d2d7, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/182148ae60634e019470f59bbab0837d] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=34.8 K 2023-04-18 18:55:45,940 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 3542937df8644932baf310cf457aab84, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681844144459 2023-04-18 18:55:45,941 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 7cbb0bd13be54e298ce09994fcd1d2d7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681844144514 2023-04-18 18:55:45,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:45,942 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 182148ae60634e019470f59bbab0837d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681844145136 2023-04-18 18:55:45,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:45,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:45,944 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:45,944 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:45,944 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:45,944 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:45,944 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,945 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,947 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,947 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:45,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,948 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,948 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:45,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:45,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:45,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:45,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:45,969 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:45,969 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:45,969 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:45,970 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#C#compaction#397 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:45,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:45,972 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=149.83 KB heapSize=420.53 KB 2023-04-18 18:55:45,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:45,973 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:45,973 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:45,973 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:45,973 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:45,973 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:45,976 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:45,976 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:45,976 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:45,977 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:45,977 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:45,977 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:45,977 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:45,977 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:45,977 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,981 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:45,981 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:45,984 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,984 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,984 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,984 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:45,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:45,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844205977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:45,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844205977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:45,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844205980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:45,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:45,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74564a00[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:45,987 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:45,987 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,988 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:45,988 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:45,989 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:45,989 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:45,989 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,989 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:45,990 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:45,990 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:45,990 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:45,990 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:45,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:45,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:45,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:45,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,992 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/69ab700726de43639d4c881f7866d5b6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/69ab700726de43639d4c881f7866d5b6 2023-04-18 18:55:45,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844205981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:45,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:45,995 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:45,995 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:45,995 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:45,995 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:45,996 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:45,997 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/C of 814b3c470b4969ca1c95de968f5dec66 into 69ab700726de43639d4c881f7866d5b6(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:45,997 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:45,997 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/C, priority=13, startTime=1681844145893; duration=0sec 2023-04-18 18:55:45,997 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:45,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:45,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844205987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:45,999 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,999 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:45,999 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:45,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:46,000 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,001 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,006 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,006 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,006 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,006 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:46,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:46,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:46,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:46,006 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:46,006 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:46,006 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,006 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,006 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,006 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,007 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:46,007 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:46,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:46,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:46,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:46,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:46,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:46,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:46,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844206086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:46,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844206087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:46,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844206093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,107 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:46,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:46,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:46,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:46,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:46,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:46,110 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:46,110 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:46,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844206099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,110 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:46,110 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:46,111 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,111 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,112 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:46,112 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,113 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,113 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:46,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:46,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:46,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:46,121 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:46,121 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:46,121 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:46,124 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:46,124 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:46,124 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:46,124 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:46,124 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:46,124 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:46,124 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:46,124 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:46,124 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,124 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:46,124 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:46,136 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:46,136 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,137 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,137 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:46,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:46,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a0fce42[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:46,138 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:46,138 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,140 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,140 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:46,140 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:46,140 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:46,140 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,141 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:46,145 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:46,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:46,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:46,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:46,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:46,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:46,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:46,151 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:46,151 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:46,151 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:46,151 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:46,151 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:46,152 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,153 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:46,153 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:46,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:46,153 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:46,154 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:46,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,157 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:46,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:46,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:46,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:46,158 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:46,158 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:46,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:46,158 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,158 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,158 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,158 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,158 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,158 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,158 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,159 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:46,159 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:46,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:46,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:46,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:46,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:46,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:46,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844206187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,258 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:46,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:46,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:46,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:46,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:46,270 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:46,270 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:46,270 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:46,270 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:46,270 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,270 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:46,273 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:46,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:46,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:46,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:46,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:46,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:46,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:46,279 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:46,279 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:46,279 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:46,279 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:46,279 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:46,279 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:46,279 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:46,279 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:46,279 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,279 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:46,279 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:46,286 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,286 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:46,286 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,286 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:46,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:46,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6321607f[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:46,288 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:46,288 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,289 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,289 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:46,290 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:46,289 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,290 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:46,290 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:46,292 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:46,292 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:46,292 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:46,292 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:46,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:46,314 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:46,314 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:46,314 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:46,314 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:46,314 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:46,315 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,315 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:46,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,316 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:46,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:46,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:46,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844206309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:46,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:46,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:46,319 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844206311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844206310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:46,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844206313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:46,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,322 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:46,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,327 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,327 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,327 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,327 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,327 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,327 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:46,327 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,327 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:46,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:46,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:46,327 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,327 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:46,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:46,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:46,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:46,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:46,328 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:46,328 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:46,334 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/0ad9257350a1434d8bb67a49a710beec as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/0ad9257350a1434d8bb67a49a710beec 2023-04-18 18:55:46,341 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/A of 814b3c470b4969ca1c95de968f5dec66 into 0ad9257350a1434d8bb67a49a710beec(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:46,341 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:46,341 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/A, priority=13, startTime=1681844145892; duration=0sec 2023-04-18 18:55:46,341 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:46,408 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/922b3527b0334751904dfc0f5375f805 2023-04-18 18:55:46,428 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:46,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:46,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:46,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:46,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:46,432 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:46,432 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:46,432 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:46,432 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:46,433 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,433 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,435 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:46,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,436 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:46,436 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,436 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:46,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:46,445 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:46,445 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:46,445 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:46,468 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:46,468 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:46,473 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:46,473 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:46,473 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:46,473 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:46,473 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:46,473 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:46,473 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,484 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:46,484 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:46,486 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,486 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:46,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:46,486 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,486 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:46,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:46,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:46,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:46,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:46,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6db1749e[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:46,488 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:46,488 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,489 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,489 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:46,489 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:46,490 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:46,490 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,490 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:46,492 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,492 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:46,492 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:46,492 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:46,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:46,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:46,495 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:46,495 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:46,495 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:46,495 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:46,495 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:46,497 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,497 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:46,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,497 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:46,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:46,498 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:46,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,508 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,510 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:46,511 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:46,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:46,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:46,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:46,511 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:46,511 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:46,511 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,511 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,511 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,511 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,511 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,511 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,511 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:46,511 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:46,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:46,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:46,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:46,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844206495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,611 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:46,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:46,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:46,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:46,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844206618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:46,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844206621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:46,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844206622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:46,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:46,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844206622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:46,634 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:46,634 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:46,634 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:46,634 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:46,634 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,634 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,635 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:46,635 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:46,636 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,636 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:46,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:46,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:46,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:46,648 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:46,648 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:46,648 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:46,660 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:46,660 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:46,664 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:46,664 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:46,664 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:46,665 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:46,665 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:46,665 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:46,665 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,665 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:46,665 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:46,667 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:46,667 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,671 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,671 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:46,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:46,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:46,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b713562[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:46,672 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:46,672 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,675 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,675 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:46,675 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:46,675 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:46,676 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,676 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:46,677 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:46,677 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:46,677 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:46,677 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:46,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:46,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:46,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:46,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:46,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:46,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:46,680 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:46,680 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:46,680 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:46,680 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:46,680 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:46,682 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,682 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:46,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,682 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:46,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:46,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:46,687 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:46,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,689 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,695 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:46,695 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:46,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:46,696 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:46,696 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:46,696 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:46,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:46,696 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,696 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,696 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,696 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,696 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,696 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:46,696 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:46,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:46,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:46,796 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:46,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:46,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:46,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:46,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:46,802 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:46,802 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:46,802 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:46,802 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:46,803 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,803 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,804 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:46,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,806 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,807 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,807 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:46,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:46,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:46,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:46,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:46,808 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:46,808 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:46,808 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:46,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:46,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:46,812 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:46,812 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:46,812 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:46,812 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:46,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:46,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:46,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:46,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:46,814 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,814 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:46,815 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,815 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:46,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:46,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:46,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:46,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f988390[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:46,818 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:46,818 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,819 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,819 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,819 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:46,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:46,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:46,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:46,825 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:46,825 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,825 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:46,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:46,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:46,825 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:46,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:46,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:46,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:46,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:46,830 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:46,830 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:46,830 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:46,830 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:46,830 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:46,831 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,831 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:46,831 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:46,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:46,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:46,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:46,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:46,832 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:46,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,834 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:46,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:46,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,839 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,839 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,839 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,839 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,839 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,839 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:46,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:46,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:46,839 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:46,839 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:46,839 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:46,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:46,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:46,839 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:46,839 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:46,839 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:46,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:46,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:46,886 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/fd48d95314c647f383ddc41167f12b61 2023-04-18 18:55:46,908 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/b481be1090bd446294eeac809fe27d3d 2023-04-18 18:55:46,913 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/922b3527b0334751904dfc0f5375f805 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/922b3527b0334751904dfc0f5375f805 2023-04-18 18:55:46,920 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/922b3527b0334751904dfc0f5375f805, entries=200, sequenceid=97, filesize=13.9 K 2023-04-18 18:55:46,921 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/fd48d95314c647f383ddc41167f12b61 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/fd48d95314c647f383ddc41167f12b61 2023-04-18 18:55:46,925 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/fd48d95314c647f383ddc41167f12b61, entries=150, sequenceid=97, filesize=11.6 K 2023-04-18 18:55:46,926 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/b481be1090bd446294eeac809fe27d3d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/b481be1090bd446294eeac809fe27d3d 2023-04-18 18:55:46,930 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/b481be1090bd446294eeac809fe27d3d, entries=150, sequenceid=97, filesize=11.6 K 2023-04-18 18:55:46,931 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~149.83 KB/153430, heapSize ~420.48 KB/430576, currentSize=33.54 KB/34350 for 814b3c470b4969ca1c95de968f5dec66 in 959ms, sequenceid=97, compaction requested=false 2023-04-18 18:55:46,931 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:46,940 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:46,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:46,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:46,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:46,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:46,943 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:46,943 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:46,943 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:46,943 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:46,944 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,945 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,946 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,946 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:46,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:46,946 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:46,946 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:46,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:46,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:46,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:46,948 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:46,948 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:46,948 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:46,952 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:46,952 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:46,952 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:46,952 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:46,952 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-18 18:55:46,952 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:46,952 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:46,952 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:46,952 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:46,953 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:46,953 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:46,974 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-18 18:55:46,994 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/773ece3b323e46b1bc8e3f57e5c52dcc 2023-04-18 18:55:47,044 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/7ad77f28fea74842b30fcd8e8859a51f 2023-04-18 18:55:47,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:47,048 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:47,086 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/f2cc4126afd047de9323655442999a2a 2023-04-18 18:55:47,093 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/773ece3b323e46b1bc8e3f57e5c52dcc as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/773ece3b323e46b1bc8e3f57e5c52dcc 2023-04-18 18:55:47,097 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/773ece3b323e46b1bc8e3f57e5c52dcc, entries=150, sequenceid=107, filesize=11.6 K 2023-04-18 18:55:47,098 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/7ad77f28fea74842b30fcd8e8859a51f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/7ad77f28fea74842b30fcd8e8859a51f 2023-04-18 18:55:47,114 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/7ad77f28fea74842b30fcd8e8859a51f, entries=150, sequenceid=107, filesize=11.6 K 2023-04-18 18:55:47,115 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/f2cc4126afd047de9323655442999a2a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f2cc4126afd047de9323655442999a2a 2023-04-18 18:55:47,126 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f2cc4126afd047de9323655442999a2a, entries=150, sequenceid=107, filesize=11.6 K 2023-04-18 18:55:47,127 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=67.09 KB/68700 for 814b3c470b4969ca1c95de968f5dec66 in 175ms, sequenceid=107, compaction requested=true 2023-04-18 18:55:47,127 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:47,127 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:47,127 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:47,127 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:47,127 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,128 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:47,128 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:47,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:47,135 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-18 18:55:47,135 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,135 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:47,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:47,136 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,136 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:47,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:47,136 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:47,136 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:47,136 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:47,136 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:47,136 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:47,136 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:47,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:47,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57d23e84[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:47,144 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:47,144 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,145 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:47,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:47,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:47,146 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,146 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:47,148 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:47,148 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:47,148 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:47,148 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:47,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:47,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:47,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:47,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:47,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:47,152 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:47,152 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:47,153 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:47,153 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:47,153 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:47,154 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,154 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:47,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,154 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:47,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:47,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:47,155 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,157 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:47,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,160 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,160 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:47,160 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:47,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:47,160 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:47,160 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:47,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:47,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:47,160 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:47,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:47,160 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,160 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,160 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,160 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:47,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:47,161 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:47,161 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:47,183 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/3f14261f6f0c4190a3b2ad4be86bc5c0 2023-04-18 18:55:47,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844207197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844207202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844207203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844207204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844207205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,261 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:47,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:47,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:47,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:47,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:47,266 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:47,266 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:47,266 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:47,266 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:47,266 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,266 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:47,268 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,268 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:47,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:47,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:47,268 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,268 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:47,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:47,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:47,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:47,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:47,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:47,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:47,270 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:47,270 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:47,271 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:47,273 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:47,273 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:47,273 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:47,273 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:47,273 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:47,273 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:47,273 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:47,273 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:47,273 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,273 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:47,273 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:47,275 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,275 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:47,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,276 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:47,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:47,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3121a564[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:47,279 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:47,279 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,280 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,281 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,281 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:47,281 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:47,281 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:47,281 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:47,283 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:47,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:47,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:47,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:47,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:47,285 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:47,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,285 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:47,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:47,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:47,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:47,288 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:47,288 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:47,288 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:47,288 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:47,288 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:47,290 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,290 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:47,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:47,290 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:47,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:47,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:47,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:47,291 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:47,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,295 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,297 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:47,297 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:47,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:47,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:47,298 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:47,298 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:47,297 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:47,298 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:47,298 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,298 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,298 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,298 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,297 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:47,298 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:47,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:47,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:47,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:47,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844207309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844207314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844207316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844207317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844207317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,398 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:47,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:47,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:47,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:47,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:47,404 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:47,404 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:47,404 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:47,404 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:47,405 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,405 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:47,406 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,406 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:47,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:47,406 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,406 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:47,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:47,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:47,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:47,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:47,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:47,409 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:47,409 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:47,409 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:47,412 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:47,412 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:47,412 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:47,412 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:47,412 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:47,412 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:47,412 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:47,412 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:47,412 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,412 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:47,412 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:47,415 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:47,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:47,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:47,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,415 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:47,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:47,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:47,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:47,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13d94c07[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:47,419 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:47,420 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,421 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,421 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:47,421 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:47,421 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,421 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:47,421 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:47,423 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:47,423 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:47,423 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:47,423 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:47,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:47,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:47,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:47,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:47,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:47,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:47,426 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:47,426 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:47,426 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:47,426 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:47,426 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:47,428 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,428 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:47,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,428 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:47,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:47,430 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,431 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:47,437 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:47,437 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:47,437 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,437 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:47,437 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:47,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:47,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:47,437 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:47,437 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,437 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,437 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:47,438 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:47,438 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:47,438 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:47,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:47,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844207517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844207523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844207524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844207524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844207525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,538 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:47,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:47,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:47,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:47,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:47,542 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:47,542 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:47,542 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:47,542 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:47,542 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,542 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:47,543 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,543 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:47,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:47,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:47,544 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,544 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:47,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:47,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:47,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:47,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:47,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:47,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:47,548 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:47,548 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:47,548 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:47,550 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:47,550 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:47,550 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:47,550 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:47,550 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:47,550 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:47,550 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:47,550 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:47,550 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,550 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:47,550 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:47,552 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:47,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:47,552 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:47,553 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,553 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:47,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:47,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68a6d9ef[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:47,555 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:47,555 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,557 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,557 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:47,557 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:47,557 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:47,557 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,557 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:47,565 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,565 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:47,566 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:47,566 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:47,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:47,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:47,569 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:47,569 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:47,569 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:47,569 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:47,569 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:47,571 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,571 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:47,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,571 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:47,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,572 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,573 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,577 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,577 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:47,577 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:47,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:47,577 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:47,577 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:47,577 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:47,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:47,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:47,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:47,577 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,578 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,578 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:47,578 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,578 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:47,578 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:47,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:47,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:47,602 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/8896a7be22ed4ad697da3d30da54a7a4 2023-04-18 18:55:47,643 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/1abcb70242c448ea83daf3fc2825fca8 2023-04-18 18:55:47,647 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/3f14261f6f0c4190a3b2ad4be86bc5c0 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/3f14261f6f0c4190a3b2ad4be86bc5c0 2023-04-18 18:55:47,651 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/3f14261f6f0c4190a3b2ad4be86bc5c0, entries=150, sequenceid=122, filesize=11.6 K 2023-04-18 18:55:47,652 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/8896a7be22ed4ad697da3d30da54a7a4 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8896a7be22ed4ad697da3d30da54a7a4 2023-04-18 18:55:47,656 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8896a7be22ed4ad697da3d30da54a7a4, entries=150, sequenceid=122, filesize=11.6 K 2023-04-18 18:55:47,657 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/1abcb70242c448ea83daf3fc2825fca8 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/1abcb70242c448ea83daf3fc2825fca8 2023-04-18 18:55:47,661 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/1abcb70242c448ea83daf3fc2825fca8, entries=150, sequenceid=122, filesize=11.6 K 2023-04-18 18:55:47,662 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 814b3c470b4969ca1c95de968f5dec66 in 527ms, sequenceid=122, compaction requested=true 2023-04-18 18:55:47,662 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:47,663 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:47,663 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:55:47,663 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:47,663 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:55:47,664 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:47,669 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:55:47,669 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/B is initiating minor compaction (all files) 2023-04-18 18:55:47,669 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/B in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:47,669 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8f08bb6524804e65a439c2266cf5e937, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/fd48d95314c647f383ddc41167f12b61, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/7ad77f28fea74842b30fcd8e8859a51f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8896a7be22ed4ad697da3d30da54a7a4] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=46.5 K 2023-04-18 18:55:47,670 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49987 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:55:47,670 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/A is initiating minor compaction (all files) 2023-04-18 18:55:47,670 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/A in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:47,670 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/0ad9257350a1434d8bb67a49a710beec, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/922b3527b0334751904dfc0f5375f805, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/773ece3b323e46b1bc8e3f57e5c52dcc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/3f14261f6f0c4190a3b2ad4be86bc5c0] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=48.8 K 2023-04-18 18:55:47,670 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 8f08bb6524804e65a439c2266cf5e937, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681844145136 2023-04-18 18:55:47,671 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 0ad9257350a1434d8bb67a49a710beec, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681844145136 2023-04-18 18:55:47,671 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting fd48d95314c647f383ddc41167f12b61, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681844145862 2023-04-18 18:55:47,671 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 922b3527b0334751904dfc0f5375f805, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681844145844 2023-04-18 18:55:47,671 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 7ad77f28fea74842b30fcd8e8859a51f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1681844145976 2023-04-18 18:55:47,672 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 773ece3b323e46b1bc8e3f57e5c52dcc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1681844145976 2023-04-18 18:55:47,672 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 8896a7be22ed4ad697da3d30da54a7a4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1681844147113 2023-04-18 18:55:47,672 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 3f14261f6f0c4190a3b2ad4be86bc5c0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1681844147113 2023-04-18 18:55:47,678 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:47,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:47,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:47,683 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#A#compaction#407 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:47,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:47,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:47,688 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:47,688 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:47,688 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:47,688 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:47,689 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,689 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:47,690 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:47,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:47,690 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:47,690 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,690 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:47,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:47,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:47,694 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#B#compaction#408 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:47,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:47,700 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:47,700 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:47,700 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:47,718 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:47,718 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:47,719 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:47,719 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:47,719 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-18 18:55:47,720 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:47,720 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:47,720 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:47,720 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:47,720 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:47,720 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:47,734 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/e614f91ce9764f329255e831aad8a116 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/e614f91ce9764f329255e831aad8a116 2023-04-18 18:55:47,738 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/B of 814b3c470b4969ca1c95de968f5dec66 into e614f91ce9764f329255e831aad8a116(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:47,738 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:47,739 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/B, priority=12, startTime=1681844147663; duration=0sec 2023-04-18 18:55:47,739 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:47,739 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:55:47,741 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:55:47,741 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/C is initiating minor compaction (all files) 2023-04-18 18:55:47,741 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/C in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:47,741 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/69ab700726de43639d4c881f7866d5b6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/b481be1090bd446294eeac809fe27d3d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f2cc4126afd047de9323655442999a2a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/1abcb70242c448ea83daf3fc2825fca8] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=46.5 K 2023-04-18 18:55:47,742 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 69ab700726de43639d4c881f7866d5b6, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681844145136 2023-04-18 18:55:47,742 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b481be1090bd446294eeac809fe27d3d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681844145862 2023-04-18 18:55:47,744 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting f2cc4126afd047de9323655442999a2a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1681844145976 2023-04-18 18:55:47,744 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 1abcb70242c448ea83daf3fc2825fca8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1681844147113 2023-04-18 18:55:47,746 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/04e07920efd9424da834ca6e9f33461e 2023-04-18 18:55:47,758 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#C#compaction#410 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:47,785 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/620785e7930e4e16a578dc4351a8f001 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/620785e7930e4e16a578dc4351a8f001 2023-04-18 18:55:47,792 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/C of 814b3c470b4969ca1c95de968f5dec66 into 620785e7930e4e16a578dc4351a8f001(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:47,792 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:47,792 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/C, priority=12, startTime=1681844147663; duration=0sec 2023-04-18 18:55:47,793 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:47,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:47,829 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:47,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844207844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844207844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844207845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844207846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844207847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844207955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844207955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844207956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844207956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:47,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:47,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844207956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,114 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/d9ea1679842b411bbf360471dd912737 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/d9ea1679842b411bbf360471dd912737 2023-04-18 18:55:48,118 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/A of 814b3c470b4969ca1c95de968f5dec66 into d9ea1679842b411bbf360471dd912737(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:48,118 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:48,118 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/A, priority=12, startTime=1681844147663; duration=0sec 2023-04-18 18:55:48,119 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:48,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:48,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844208161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:48,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844208162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:48,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:48,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844208163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844208162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:48,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844208163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,174 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/ec23ef2a32f54dbf9741600d7077fc65 2023-04-18 18:55:48,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:48,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844208467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:48,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844208471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:48,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844208471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:48,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844208472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:48,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844208472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,586 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/e9d2b44108a6464ebca004909ee94d17 2023-04-18 18:55:48,591 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/04e07920efd9424da834ca6e9f33461e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/04e07920efd9424da834ca6e9f33461e 2023-04-18 18:55:48,595 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/04e07920efd9424da834ca6e9f33461e, entries=150, sequenceid=139, filesize=11.8 K 2023-04-18 18:55:48,596 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/ec23ef2a32f54dbf9741600d7077fc65 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ec23ef2a32f54dbf9741600d7077fc65 2023-04-18 18:55:48,599 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ec23ef2a32f54dbf9741600d7077fc65, entries=150, sequenceid=139, filesize=11.8 K 2023-04-18 18:55:48,600 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/e9d2b44108a6464ebca004909ee94d17 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/e9d2b44108a6464ebca004909ee94d17 2023-04-18 18:55:48,605 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/e9d2b44108a6464ebca004909ee94d17, entries=150, sequenceid=139, filesize=11.8 K 2023-04-18 18:55:48,606 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 814b3c470b4969ca1c95de968f5dec66 in 887ms, sequenceid=139, compaction requested=false 2023-04-18 18:55:48,606 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:48,606 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:48,606 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:48,606 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:48,606 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,606 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:48,606 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:48,608 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,609 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:48,609 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:48,609 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:48,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:48,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e46ec61[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:48,611 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:48,611 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:48,612 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:48,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:48,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:48,613 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:48,613 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:48,613 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:48,613 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,613 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:48,614 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:48,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:48,615 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:48,615 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:48,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:48,615 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:48,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:48,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:48,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:48,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:48,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:48,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:48,618 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:48,618 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:48,618 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:48,618 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:48,618 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:48,619 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:48,619 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:48,619 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:48,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:48,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:48,620 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:48,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,629 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,629 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:48,629 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:48,629 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,629 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:48,629 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:48,629 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:48,629 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:48,629 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:48,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:48,629 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:48,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:48,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:48,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:48,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:48,630 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:48,630 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:48,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:48,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:48,730 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:48,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:48,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:48,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:48,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:48,733 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:48,733 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:48,733 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:48,733 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:48,734 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:48,734 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:48,735 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,735 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:48,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:48,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:48,735 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,735 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:48,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:48,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:48,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:48,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:48,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:48,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:48,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:48,744 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:48,744 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:48,744 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:48,744 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:48,744 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-18 18:55:48,744 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:48,744 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:48,744 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:48,745 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:48,745 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:48,745 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:48,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:48,979 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:48,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:48,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844208990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:48,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844208991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:48,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:48,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844208995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:49,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:49,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844208997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:49,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:49,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844208999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:49,107 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:49,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844209100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:49,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:49,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:49,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844209100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:49,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844209100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:49,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:49,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844209103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:49,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:49,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844209109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:49,173 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/875b060be5124ef8be85717a8b33712f 2023-04-18 18:55:49,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:49,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844209309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:49,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:49,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844209309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:49,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:49,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844209310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:49,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:49,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844209310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:49,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:49,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844209312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:49,585 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/76c19dbb6d544fb9a3bfa075f8e71cde 2023-04-18 18:55:49,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:49,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844209616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:49,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:49,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844209617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:49,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:49,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844209618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:49,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:49,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844209618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:49,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:49,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844209620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:49,995 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/aa43993c73e34bf98cfc7508810b232c 2023-04-18 18:55:50,000 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/875b060be5124ef8be85717a8b33712f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/875b060be5124ef8be85717a8b33712f 2023-04-18 18:55:50,004 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/875b060be5124ef8be85717a8b33712f, entries=150, sequenceid=157, filesize=11.8 K 2023-04-18 18:55:50,005 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/76c19dbb6d544fb9a3bfa075f8e71cde as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/76c19dbb6d544fb9a3bfa075f8e71cde 2023-04-18 18:55:50,014 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/76c19dbb6d544fb9a3bfa075f8e71cde, entries=150, sequenceid=157, filesize=11.8 K 2023-04-18 18:55:50,014 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/aa43993c73e34bf98cfc7508810b232c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/aa43993c73e34bf98cfc7508810b232c 2023-04-18 18:55:50,018 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/aa43993c73e34bf98cfc7508810b232c, entries=150, sequenceid=157, filesize=11.8 K 2023-04-18 18:55:50,019 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 814b3c470b4969ca1c95de968f5dec66 in 1275ms, sequenceid=157, compaction requested=true 2023-04-18 18:55:50,019 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:50,019 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:50,020 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:50,021 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:50,022 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:50,022 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,022 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:50,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:50,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12f07648[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:50,023 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:50,024 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,025 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,025 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:50,025 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:50,025 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:50,026 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,026 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:50,029 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:50,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:50,032 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:50,032 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:50,032 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:50,032 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:50,032 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:50,036 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,036 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:50,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:50,038 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,039 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,051 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,051 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:50,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:50,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:50,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:50,051 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:50,051 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:50,052 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:50,052 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:50,051 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,052 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,052 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,052 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,052 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,052 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:50,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:50,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:50,133 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-18 18:55:50,134 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:50,134 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:50,134 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:50,135 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:50,135 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:50,135 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:50,161 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:50,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:50,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:50,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:50,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844210164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:50,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:50,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:50,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:50,180 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:50,180 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:50,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844210166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844210165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,180 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:50,180 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:50,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:50,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:50,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844210173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844210172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,182 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,182 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,187 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:50,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:50,187 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,188 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:50,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:50,189 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:50,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:50,189 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:50,189 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:50,189 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:50,189 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:50,189 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:50,190 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:50,190 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:50,190 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:50,190 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:50,190 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:50,190 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:50,190 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,190 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:50,190 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:50,192 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,192 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:50,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:50,193 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,193 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:50,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:50,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:50,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b10b965[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:50,194 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:50,194 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,197 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,197 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:50,197 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:50,197 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:50,198 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:50,198 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:50,198 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:50,199 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,199 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:50,199 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:50,199 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:50,199 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:50,199 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:50,199 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:50,199 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:50,202 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,202 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:50,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:50,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:50,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,204 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:50,205 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:50,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,206 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:50,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,209 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,209 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,209 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,209 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:50,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:50,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:50,209 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:50,209 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:50,209 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,210 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,210 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:50,210 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:50,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:50,210 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,210 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:50,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:50,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:50,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844210277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:50,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844210282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:50,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844210282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:50,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844210282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:50,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844210283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,310 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:50,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:50,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:50,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:50,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:50,314 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:50,314 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:50,314 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:50,314 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:50,315 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,315 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,318 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:50,318 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:50,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:50,320 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,320 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:50,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:50,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:50,321 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:50,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:50,321 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:50,321 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:50,321 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:50,321 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:50,321 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:50,321 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:50,321 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:50,322 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:50,322 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:50,322 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:50,322 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:50,322 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,322 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:50,322 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:50,323 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,324 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,324 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:50,324 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:50,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:50,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:50,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:50,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:50,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:50,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15277849[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:50,326 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:50,326 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,329 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,329 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:50,329 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,329 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:50,329 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:50,329 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:50,331 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:50,331 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:50,331 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,331 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:50,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:50,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:50,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:50,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:50,336 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:50,336 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:50,336 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:50,336 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:50,336 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:50,338 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,338 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,338 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:50,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:50,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,339 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,340 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,342 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,342 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,342 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,342 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:50,342 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:50,342 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:50,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:50,342 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,343 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,343 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,342 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:50,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:50,343 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:50,343 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:50,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:50,443 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:50,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:50,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:50,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:50,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:50,446 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:50,446 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:50,446 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:50,447 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:50,447 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,447 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,452 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:50,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:50,452 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,453 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,453 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:50,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:50,454 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:50,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:50,454 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:50,454 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:50,454 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:50,454 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:50,454 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:50,454 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:50,454 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:50,454 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:50,454 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:50,454 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:50,454 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:50,454 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,454 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:50,454 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:50,456 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,456 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:50,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:50,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:50,457 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,457 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:50,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:50,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:50,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:50,458 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:50,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c9bbb68[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:50,458 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,459 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,460 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:50,460 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:50,460 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:50,461 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,461 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:50,461 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,461 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:50,461 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:50,462 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:50,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:50,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:50,465 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:50,466 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:50,466 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:50,466 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:50,466 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:50,468 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,468 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,468 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:50,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:50,469 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,471 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,474 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,474 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,474 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,474 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,474 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,474 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:50,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:50,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:50,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:50,474 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:50,474 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:50,475 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:50,475 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:50,474 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,475 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:50,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:50,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844210484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:50,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844210485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:50,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844210488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:50,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844210488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:50,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844210488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,574 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/7352789bb5e74a2ab530e3bce7f9150d 2023-04-18 18:55:50,575 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:50,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:50,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:50,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:50,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:50,579 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:50,579 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:50,579 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:50,579 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:50,581 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,581 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,585 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,585 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:50,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,586 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,586 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:50,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:50,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:50,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:50,593 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:50,593 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:50,593 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:50,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:50,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:50,603 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:50,603 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:50,603 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:50,603 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:50,603 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:50,604 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:50,604 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,608 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:50,608 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:50,611 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,611 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:50,611 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,612 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:50,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:50,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d48de85[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:50,614 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:50,614 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,615 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,616 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:50,616 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:50,616 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:50,616 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,616 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:50,617 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,617 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:50,618 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:50,618 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:50,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:50,621 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:50,621 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:50,621 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:50,621 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:50,621 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:50,623 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,623 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:50,623 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:50,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:50,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:50,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:50,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:50,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,624 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,626 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,631 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:50,631 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:50,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:50,631 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:50,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:50,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:50,631 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:50,631 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:50,631 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:50,631 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,631 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,632 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:50,632 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,632 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,632 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,731 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:50,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:50,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:50,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:50,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:50,736 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:50,737 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:50,737 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:50,737 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:50,737 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,737 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,738 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:50,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,739 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,739 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,739 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:50,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:50,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:50,740 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:50,740 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:50,740 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:50,740 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:50,740 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:50,740 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:50,740 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:50,740 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:50,740 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:50,740 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:50,740 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:50,740 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,740 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:50,740 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:50,742 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,742 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,742 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:50,742 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:50,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:50,744 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:50,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@535d97d8[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:50,744 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,745 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,746 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:50,746 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:50,746 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:50,746 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,746 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:50,748 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,748 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:50,748 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:50,748 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:50,750 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:50,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:50,750 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:50,750 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:50,751 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:50,751 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:50,752 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,752 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,752 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:50,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:50,753 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,755 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,757 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,757 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,757 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,757 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,757 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,757 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:50,757 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,757 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,757 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:50,757 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:50,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:50,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:50,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:50,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:50,758 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:50,758 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:50,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:50,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:50,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:50,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844210793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:50,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844210793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:50,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844210793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:50,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844210794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:50,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844210794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,858 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:50,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:50,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:50,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:50,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:50,862 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:50,862 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:50,862 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:50,862 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:50,862 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,862 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,863 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,863 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:50,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:50,864 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,864 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:50,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:50,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:50,865 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:50,865 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:50,865 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:50,866 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:50,866 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:50,866 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:50,866 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:50,867 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:50,867 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:50,867 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:50,867 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:50,867 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,867 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:50,867 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:50,870 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,870 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:50,870 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:50,870 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:50,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:50,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d9658ca[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:50,872 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:50,872 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,873 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,873 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,873 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:50,873 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:50,873 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:50,873 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:50,875 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:50,875 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,875 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:50,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:50,877 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:50,877 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:50,877 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:50,877 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:50,877 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:50,879 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,879 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,879 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:50,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:50,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:50,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:50,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:50,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:50,879 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,880 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,883 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,883 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,884 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:50,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:50,884 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:50,884 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:50,884 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:50,884 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,884 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,884 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,884 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,884 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:50,884 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:50,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:50,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:50,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:50,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:50,984 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:50,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:50,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:50,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:50,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:50,988 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:50,988 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:50,988 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:50,988 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:50,988 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,988 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,990 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,990 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:50,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:50,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:50,990 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,990 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:50,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:50,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:50,991 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:50,991 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:50,991 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:50,991 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:50,991 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:50,992 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:50,992 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:50,992 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:50,992 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:50,992 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:50,992 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:50,993 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:50,993 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:50,993 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,993 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:50,993 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:50,997 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:50,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:50,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c87fceb[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:50,999 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:50,999 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,001 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,001 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:51,001 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:51,001 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,001 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:51,001 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:51,002 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:51,003 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:51,003 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:51,003 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:51,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:51,006 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:51,006 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:51,006 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:51,006 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:51,006 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:51,008 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:51,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:51,008 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,008 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:51,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:51,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:51,009 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,010 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,013 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:51,013 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:51,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:51,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:51,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:51,013 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,013 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:51,013 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:51,013 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:51,013 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:51,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:51,013 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,013 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,013 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,013 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:51,014 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:51,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:51,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:51,019 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/4152098fb1a5434cace9f89b49edb280 2023-04-18 18:55:51,033 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/1b0b03d89e294496a7404c234be15add 2023-04-18 18:55:51,039 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/7352789bb5e74a2ab530e3bce7f9150d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/7352789bb5e74a2ab530e3bce7f9150d 2023-04-18 18:55:51,042 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/7352789bb5e74a2ab530e3bce7f9150d, entries=150, sequenceid=177, filesize=11.8 K 2023-04-18 18:55:51,043 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/4152098fb1a5434cace9f89b49edb280 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/4152098fb1a5434cace9f89b49edb280 2023-04-18 18:55:51,049 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/4152098fb1a5434cace9f89b49edb280, entries=150, sequenceid=177, filesize=11.8 K 2023-04-18 18:55:51,050 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/1b0b03d89e294496a7404c234be15add as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/1b0b03d89e294496a7404c234be15add 2023-04-18 18:55:51,054 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/1b0b03d89e294496a7404c234be15add, entries=150, sequenceid=177, filesize=11.8 K 2023-04-18 18:55:51,055 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 814b3c470b4969ca1c95de968f5dec66 in 922ms, sequenceid=177, compaction requested=true 2023-04-18 18:55:51,055 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:51,055 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:51,055 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:51,055 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:55:51,056 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:55:51,056 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:51,057 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48234 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:55:51,057 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/A is initiating minor compaction (all files) 2023-04-18 18:55:51,057 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/A in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:51,057 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48234 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:55:51,057 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/d9ea1679842b411bbf360471dd912737, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/04e07920efd9424da834ca6e9f33461e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/875b060be5124ef8be85717a8b33712f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/7352789bb5e74a2ab530e3bce7f9150d] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=47.1 K 2023-04-18 18:55:51,058 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/B is initiating minor compaction (all files) 2023-04-18 18:55:51,058 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/B in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:51,058 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/e614f91ce9764f329255e831aad8a116, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ec23ef2a32f54dbf9741600d7077fc65, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/76c19dbb6d544fb9a3bfa075f8e71cde, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/4152098fb1a5434cace9f89b49edb280] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=47.1 K 2023-04-18 18:55:51,058 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting d9ea1679842b411bbf360471dd912737, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1681844147113 2023-04-18 18:55:51,058 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting e614f91ce9764f329255e831aad8a116, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1681844147113 2023-04-18 18:55:51,059 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 04e07920efd9424da834ca6e9f33461e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1681844147184 2023-04-18 18:55:51,059 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting ec23ef2a32f54dbf9741600d7077fc65, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1681844147184 2023-04-18 18:55:51,059 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 875b060be5124ef8be85717a8b33712f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681844147836 2023-04-18 18:55:51,059 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 76c19dbb6d544fb9a3bfa075f8e71cde, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681844147836 2023-04-18 18:55:51,060 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 7352789bb5e74a2ab530e3bce7f9150d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1681844150131 2023-04-18 18:55:51,060 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 4152098fb1a5434cace9f89b49edb280, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1681844150131 2023-04-18 18:55:51,069 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#A#compaction#419 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:51,074 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#B#compaction#420 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:51,098 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/6ecfa75ea86d4f6088e4b6d5f2c4cfad as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/6ecfa75ea86d4f6088e4b6d5f2c4cfad 2023-04-18 18:55:51,103 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/B of 814b3c470b4969ca1c95de968f5dec66 into 6ecfa75ea86d4f6088e4b6d5f2c4cfad(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:51,103 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:51,103 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/B, priority=12, startTime=1681844151055; duration=0sec 2023-04-18 18:55:51,103 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:51,103 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:55:51,105 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48234 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:55:51,105 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/C is initiating minor compaction (all files) 2023-04-18 18:55:51,105 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/C in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:51,105 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/620785e7930e4e16a578dc4351a8f001, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/e9d2b44108a6464ebca004909ee94d17, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/aa43993c73e34bf98cfc7508810b232c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/1b0b03d89e294496a7404c234be15add] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=47.1 K 2023-04-18 18:55:51,105 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 620785e7930e4e16a578dc4351a8f001, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1681844147113 2023-04-18 18:55:51,106 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting e9d2b44108a6464ebca004909ee94d17, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1681844147184 2023-04-18 18:55:51,106 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting aa43993c73e34bf98cfc7508810b232c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681844147836 2023-04-18 18:55:51,106 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 1b0b03d89e294496a7404c234be15add, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1681844150131 2023-04-18 18:55:51,114 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:51,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:51,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:51,122 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#C#compaction#421 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:51,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:51,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:51,128 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:51,128 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:51,128 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:51,128 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:51,129 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,129 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:51,131 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:51,131 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:51,133 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,133 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:51,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:51,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:51,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:51,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:51,140 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:51,140 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:51,140 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:51,147 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:51,147 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:51,147 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:51,147 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:51,147 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-18 18:55:51,147 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:51,148 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:51,148 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:51,149 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:51,149 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:51,149 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:51,171 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/adbb3f2b54b0454e9eac5c43e4740df4 2023-04-18 18:55:51,196 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/767210ce97bc4277bda2732ebde38889 2023-04-18 18:55:51,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:51,303 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:51,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:51,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844211325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:51,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844211326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:51,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844211327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:51,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844211328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:51,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844211337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:51,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844211438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:51,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844211438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:51,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844211438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:51,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844211440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:51,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844211450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,489 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/76b58a9ef82b4290829b3d9a0b82d57e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/76b58a9ef82b4290829b3d9a0b82d57e 2023-04-18 18:55:51,494 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/A of 814b3c470b4969ca1c95de968f5dec66 into 76b58a9ef82b4290829b3d9a0b82d57e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:51,494 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:51,494 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/A, priority=12, startTime=1681844151055; duration=0sec 2023-04-18 18:55:51,494 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:51,561 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/6c8f018ef9104262a0eab002b7725bf8 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/6c8f018ef9104262a0eab002b7725bf8 2023-04-18 18:55:51,565 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/C of 814b3c470b4969ca1c95de968f5dec66 into 6c8f018ef9104262a0eab002b7725bf8(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:51,565 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:51,565 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/C, priority=12, startTime=1681844151056; duration=0sec 2023-04-18 18:55:51,565 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:51,616 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/62ae2d8298f642c981b989126808830d 2023-04-18 18:55:51,621 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/adbb3f2b54b0454e9eac5c43e4740df4 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/adbb3f2b54b0454e9eac5c43e4740df4 2023-04-18 18:55:51,624 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/adbb3f2b54b0454e9eac5c43e4740df4, entries=150, sequenceid=190, filesize=11.8 K 2023-04-18 18:55:51,625 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/767210ce97bc4277bda2732ebde38889 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/767210ce97bc4277bda2732ebde38889 2023-04-18 18:55:51,629 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/767210ce97bc4277bda2732ebde38889, entries=150, sequenceid=190, filesize=11.8 K 2023-04-18 18:55:51,630 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/62ae2d8298f642c981b989126808830d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/62ae2d8298f642c981b989126808830d 2023-04-18 18:55:51,635 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/62ae2d8298f642c981b989126808830d, entries=150, sequenceid=190, filesize=11.8 K 2023-04-18 18:55:51,636 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 814b3c470b4969ca1c95de968f5dec66 in 489ms, sequenceid=190, compaction requested=false 2023-04-18 18:55:51,636 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:51,636 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,637 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:51,637 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:51,638 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,638 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:51,639 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,639 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:51,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:51,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7760a308[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:51,641 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:51,641 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,643 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:51,643 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,643 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:51,646 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,646 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:51,646 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:51,646 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:51,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:51,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:51,650 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:51,650 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:51,650 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:51,650 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:51,650 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:51,653 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,653 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:51,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,653 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:51,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:51,653 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-18 18:55:51,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:51,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1681844151323, max timestamp=1681844151651Num uniques 859; 2023-04-18 18:55:51,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7140, suffix heap size=117064,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=156 2023-04-18 18:55:51,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844151323, max timestamp=1681844151651Num uniques 859; 2023-04-18 18:55:51,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844151323, max timestamp=1681844151651Num uniques 859; 2023-04-18 18:55:51,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:55:51,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:51,655 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:51,655 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:51,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:55:51,655 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,655 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,655 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:51,656 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:51,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:51,657 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:51,660 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:51,660 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:51,660 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:51,660 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:51,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:51,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:51,660 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:51,660 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:51,660 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:51,661 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:51,661 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,661 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,661 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,661 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:51,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:51,671 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/3bad581c53ba47008ef21c65b90be0fc 2023-04-18 18:55:51,685 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/5ebe0bc8a6614df4b063e36ac3a462a1 2023-04-18 18:55:51,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681844151679, max timestamp=1681844151692Num uniques 859; 2023-04-18 18:55:51,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 18:55:51,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681844151679, max timestamp=1681844151692Num uniques 859; 2023-04-18 18:55:51,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681844151679, max timestamp=1681844151692Num uniques 859; 2023-04-18 18:55:51,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 18:55:51,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 18:55:51,702 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.29 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/f90a088424964fb692056d4be83c242c 2023-04-18 18:55:51,706 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/3bad581c53ba47008ef21c65b90be0fc as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/3bad581c53ba47008ef21c65b90be0fc 2023-04-18 18:55:51,709 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/3bad581c53ba47008ef21c65b90be0fc, entries=250, sequenceid=215, filesize=16.4 K 2023-04-18 18:55:51,710 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/5ebe0bc8a6614df4b063e36ac3a462a1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/5ebe0bc8a6614df4b063e36ac3a462a1 2023-04-18 18:55:51,714 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/5ebe0bc8a6614df4b063e36ac3a462a1, entries=150, sequenceid=215, filesize=11.8 K 2023-04-18 18:55:51,715 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/f90a088424964fb692056d4be83c242c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f90a088424964fb692056d4be83c242c 2023-04-18 18:55:51,728 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f90a088424964fb692056d4be83c242c, entries=150, sequenceid=215, filesize=11.8 K 2023-04-18 18:55:51,729 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.12 KB/41085, heapSize ~105.73 KB/108264, currentSize=127.47 KB/130530 for 814b3c470b4969ca1c95de968f5dec66 in 76ms, sequenceid=215, compaction requested=true 2023-04-18 18:55:51,729 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:51,729 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:51,729 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:51,729 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:51,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:51,729 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:51,730 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:51,730 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-18 18:55:51,731 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41264 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:51,731 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/A is initiating minor compaction (all files) 2023-04-18 18:55:51,731 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:51,731 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/A in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:51,731 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:51,732 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/76b58a9ef82b4290829b3d9a0b82d57e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/adbb3f2b54b0454e9eac5c43e4740df4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/3bad581c53ba47008ef21c65b90be0fc] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=40.3 K 2023-04-18 18:55:51,732 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:51,732 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:51,732 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:51,732 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:51,732 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 76b58a9ef82b4290829b3d9a0b82d57e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1681844150131 2023-04-18 18:55:51,732 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:51,732 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/B is initiating minor compaction (all files) 2023-04-18 18:55:51,733 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/B in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:51,733 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/6ecfa75ea86d4f6088e4b6d5f2c4cfad, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/767210ce97bc4277bda2732ebde38889, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/5ebe0bc8a6614df4b063e36ac3a462a1] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=35.6 K 2023-04-18 18:55:51,733 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting adbb3f2b54b0454e9eac5c43e4740df4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1681844150142 2023-04-18 18:55:51,733 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 6ecfa75ea86d4f6088e4b6d5f2c4cfad, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1681844150131 2023-04-18 18:55:51,733 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 3bad581c53ba47008ef21c65b90be0fc, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681844151327 2023-04-18 18:55:51,734 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 767210ce97bc4277bda2732ebde38889, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1681844150142 2023-04-18 18:55:51,734 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 5ebe0bc8a6614df4b063e36ac3a462a1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681844151327 2023-04-18 18:55:51,754 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#B#compaction#429 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:51,756 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#A#compaction#430 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:51,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:51,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844211746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:51,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844211747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:51,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844211747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:51,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844211749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:51,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844211757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,761 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:51,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:51,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:51,776 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/87b30d6f0b524e469669bf38157f6bf1 2023-04-18 18:55:51,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:51,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:51,776 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:51,776 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:51,776 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:51,777 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:51,777 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,777 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:51,780 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:51,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:51,780 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,781 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,781 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:51,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:51,782 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:51,782 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:51,782 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:51,782 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:51,782 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:51,782 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:51,784 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:51,784 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:51,784 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:51,784 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:51,784 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:51,785 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:51,785 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:51,785 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:51,785 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,785 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:51,785 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:51,787 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:51,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:51,789 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:51,790 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,790 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:51,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:51,790 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/9ad2742c6a774191b0ef8f27a66a62a2 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/9ad2742c6a774191b0ef8f27a66a62a2 2023-04-18 18:55:51,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:51,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:51,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48bc096b[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:51,792 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:51,792 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,794 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,794 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:51,794 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:51,794 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:51,794 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,794 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:51,795 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,795 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:51,795 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:51,796 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:51,797 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/B of 814b3c470b4969ca1c95de968f5dec66 into 9ad2742c6a774191b0ef8f27a66a62a2(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:51,797 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:51,797 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/B, priority=13, startTime=1681844151729; duration=0sec 2023-04-18 18:55:51,797 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:51,797 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:51,799 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:51,799 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/C is initiating minor compaction (all files) 2023-04-18 18:55:51,799 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/C in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:51,799 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/6c8f018ef9104262a0eab002b7725bf8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/62ae2d8298f642c981b989126808830d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f90a088424964fb692056d4be83c242c] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=35.6 K 2023-04-18 18:55:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:51,800 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 6c8f018ef9104262a0eab002b7725bf8, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1681844150131 2023-04-18 18:55:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,800 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 62ae2d8298f642c981b989126808830d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1681844150142 2023-04-18 18:55:51,800 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting f90a088424964fb692056d4be83c242c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681844151327 2023-04-18 18:55:51,802 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/9f3c3ffc0aec458da1da795caa7bcaf5 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/9f3c3ffc0aec458da1da795caa7bcaf5 2023-04-18 18:55:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:51,807 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/A of 814b3c470b4969ca1c95de968f5dec66 into 9f3c3ffc0aec458da1da795caa7bcaf5(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:51,807 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:51,807 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/A, priority=13, startTime=1681844151729; duration=0sec 2023-04-18 18:55:51,807 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:51,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:51,809 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:51,809 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#C#compaction#432 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:51,809 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:51,809 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:51,809 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:51,809 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:51,815 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,815 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,815 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:51,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:51,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:51,816 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:51,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,818 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:51,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,826 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:51,826 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:51,826 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,826 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:51,826 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:51,826 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,826 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:51,827 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:51,827 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:51,827 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:51,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:51,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:51,827 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/f977335a15324f4e8cde60ed293163f6 2023-04-18 18:55:51,827 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:51,827 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:51,846 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/47238a0f2a554909bcd2aa62bdb4d153 2023-04-18 18:55:51,850 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/87b30d6f0b524e469669bf38157f6bf1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/87b30d6f0b524e469669bf38157f6bf1 2023-04-18 18:55:51,855 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/87b30d6f0b524e469669bf38157f6bf1, entries=200, sequenceid=251, filesize=14.1 K 2023-04-18 18:55:51,856 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/f977335a15324f4e8cde60ed293163f6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/f977335a15324f4e8cde60ed293163f6 2023-04-18 18:55:51,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:51,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844211859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:51,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844211859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,867 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/f977335a15324f4e8cde60ed293163f6, entries=150, sequenceid=251, filesize=11.8 K 2023-04-18 18:55:51,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:51,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844211861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:51,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844211861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:51,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844211861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,868 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/47238a0f2a554909bcd2aa62bdb4d153 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/47238a0f2a554909bcd2aa62bdb4d153 2023-04-18 18:55:51,873 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/47238a0f2a554909bcd2aa62bdb4d153, entries=150, sequenceid=251, filesize=11.8 K 2023-04-18 18:55:51,874 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=40.25 KB/41220 for 814b3c470b4969ca1c95de968f5dec66 in 144ms, sequenceid=251, compaction requested=false 2023-04-18 18:55:51,874 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:51,927 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:51,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:51,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:51,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:51,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:51,931 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:51,931 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:51,931 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:51,931 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:51,931 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,931 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:51,932 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,933 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:51,933 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:51,933 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:51,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:51,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:51,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:51,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:51,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:51,935 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:51,935 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:51,935 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:51,936 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:51,936 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:51,936 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:51,936 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:51,936 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-18 18:55:51,936 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:51,936 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:51,936 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:51,936 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:51,937 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:51,937 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:51,953 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/cfab6d83b19842aa8590121b3e7f5fa2 2023-04-18 18:55:51,966 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/41f59f53c20d49cd8572e6e0869697d9 2023-04-18 18:55:52,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:52,074 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:52,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844152086, max timestamp=1681844152108Num uniques 859; 2023-04-18 18:55:52,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:55:52,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844152086, max timestamp=1681844152108Num uniques 859; 2023-04-18 18:55:52,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844152086, max timestamp=1681844152108Num uniques 859; 2023-04-18 18:55:52,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:55:52,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:55:52,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:52,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844212139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681844152130, max timestamp=1681844152144Num uniques 1018; 2023-04-18 18:55:52,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681844152130, max timestamp=1681844152144Num uniques 1018; 2023-04-18 18:55:52,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681844152130, max timestamp=1681844152144Num uniques 1018; 2023-04-18 18:55:52,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-18 18:55:52,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-18 18:55:52,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-18 18:55:52,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:52,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844212183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:52,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844212183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:52,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844212184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:52,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844212185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,227 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/f8a13fa737214ae1b84f35213bf333e3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f8a13fa737214ae1b84f35213bf333e3 2023-04-18 18:55:52,232 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/C of 814b3c470b4969ca1c95de968f5dec66 into f8a13fa737214ae1b84f35213bf333e3(size=12.2 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:52,232 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:52,232 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/C, priority=13, startTime=1681844151730; duration=0sec 2023-04-18 18:55:52,232 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:52,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:52,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60820 deadline: 1681844212247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:52,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60778 deadline: 1681844212286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,307 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:52,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:52,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60816 deadline: 1681844212298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60796 deadline: 1681844212298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:52,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60806 deadline: 1681844212298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,385 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/f67cf587c48b493c826cb6548975ab92 2023-04-18 18:55:52,389 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/cfab6d83b19842aa8590121b3e7f5fa2 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/cfab6d83b19842aa8590121b3e7f5fa2 2023-04-18 18:55:52,405 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/cfab6d83b19842aa8590121b3e7f5fa2, entries=150, sequenceid=262, filesize=11.9 K 2023-04-18 18:55:52,406 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/41f59f53c20d49cd8572e6e0869697d9 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/41f59f53c20d49cd8572e6e0869697d9 2023-04-18 18:55:52,428 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/41f59f53c20d49cd8572e6e0869697d9, entries=150, sequenceid=262, filesize=11.9 K 2023-04-18 18:55:52,429 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/f67cf587c48b493c826cb6548975ab92 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f67cf587c48b493c826cb6548975ab92 2023-04-18 18:55:52,441 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f67cf587c48b493c826cb6548975ab92, entries=150, sequenceid=262, filesize=11.9 K 2023-04-18 18:55:52,442 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for 814b3c470b4969ca1c95de968f5dec66 in 506ms, sequenceid=262, compaction requested=true 2023-04-18 18:55:52,442 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:52,442 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:52,442 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:52,442 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:52,442 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,444 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:52,444 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:52,445 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,446 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,446 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:52,452 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a348be[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:52,454 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:52,454 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,455 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,455 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:52,455 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:52,455 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:52,455 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,455 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:52,460 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:52,460 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:52,460 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,460 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:52,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681844152173, max timestamp=1681844152459Num uniques 1127; 2023-04-18 18:55:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:52,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681844152173, max timestamp=1681844152459Num uniques 1127; 2023-04-18 18:55:52,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-18 18:55:52,463 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=111.82 KB heapSize=311.67 KB 2023-04-18 18:55:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,463 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:52,463 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:52,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681844152173, max timestamp=1681844152459Num uniques 1127; 2023-04-18 18:55:52,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-18 18:55:52,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-18 18:55:52,463 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,463 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:52,464 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:52,464 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:52,464 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:52,464 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:52,464 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:52,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:52,464 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:52,464 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:52,467 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,467 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:52,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,468 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:52,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:52,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:52,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:52,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,470 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:52,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,472 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,475 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,475 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:52,475 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:52,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:52,475 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:52,475 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:52,475 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,475 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:52,475 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,475 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,475 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:52,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:52,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:52,476 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:52,476 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:52,482 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/31050cfe79564201af448e84127f29c3 2023-04-18 18:55:52,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681844152517, max timestamp=1681844152528Num uniques 859; 2023-04-18 18:55:52,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 18:55:52,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681844152517, max timestamp=1681844152528Num uniques 859; 2023-04-18 18:55:52,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 18:55:52,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681844152517, max timestamp=1681844152528Num uniques 859; 2023-04-18 18:55:52,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 18:55:52,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681844152552, max timestamp=1681844152564Num uniques 1118; 2023-04-18 18:55:52,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-18 18:55:52,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681844152552, max timestamp=1681844152564Num uniques 1118; 2023-04-18 18:55:52,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-18 18:55:52,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681844152552, max timestamp=1681844152564Num uniques 1118; 2023-04-18 18:55:52,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-18 18:55:52,576 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:52,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:52,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:52,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:52,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:52,581 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:52,581 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:52,581 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:52,581 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:52,581 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,581 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:52,582 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:52,582 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:52,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:52,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:52,583 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,583 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:52,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:52,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:52,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:52,585 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:52,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:52,585 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:52,585 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:52,585 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:52,585 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:52,585 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:52,585 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:52,585 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:52,585 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:52,585 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:52,585 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:52,585 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:52,585 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,586 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:52,586 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:52,588 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,588 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:52,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:52,588 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,588 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:52,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:52,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:52,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:52,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:52,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14509c05[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:52,590 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:52,590 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,591 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,591 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:52,591 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:52,591 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:52,592 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,592 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:52,593 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:52,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:52,593 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:52,593 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:52,593 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:52,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:52,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:52,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:52,595 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:52,595 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:52,595 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:52,595 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:52,595 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:52,597 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,597 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:52,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,597 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:52,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:52,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:52,598 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:52,599 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,602 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,602 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:52,602 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:52,602 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:52,603 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:52,603 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,603 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,603 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,603 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:52,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:52,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:52,603 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:52,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:52,603 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:52,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:52,603 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:52,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:52,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844212592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12681, totalHeapSize=29048, min timestamp=1681844152588, max timestamp=1681844152604Num uniques 1077; 2023-04-18 18:55:52,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49321, new segment data size=12681, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-18 18:55:52,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12677, totalHeapSize=29048, min timestamp=1681844152588, max timestamp=1681844152604Num uniques 1127; 2023-04-18 18:55:52,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51607, new segment data size=12677, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-04-18 18:55:52,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12672, totalHeapSize=29048, min timestamp=1681844152588, max timestamp=1681844152604Num uniques 1127; 2023-04-18 18:55:52,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51602, new segment data size=12672, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-04-18 18:55:52,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681844152636, max timestamp=1681844152649Num uniques 1136; 2023-04-18 18:55:52,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-18 18:55:52,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681844152636, max timestamp=1681844152649Num uniques 1136; 2023-04-18 18:55:52,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681844152636, max timestamp=1681844152649Num uniques 1136; 2023-04-18 18:55:52,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-18 18:55:52,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-18 18:55:52,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681844152690, max timestamp=1681844152694Num uniques 1095; 2023-04-18 18:55:52,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-18 18:55:52,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681844152690, max timestamp=1681844152694Num uniques 1095; 2023-04-18 18:55:52,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-18 18:55:52,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681844152690, max timestamp=1681844152694Num uniques 1095; 2023-04-18 18:55:52,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-18 18:55:52,703 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:52,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:52,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:52,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:52,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:52,707 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:52,707 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:52,707 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:52,707 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:52,708 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,708 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:52,709 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:52,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:52,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:52,709 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:52,710 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,710 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:52,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:52,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:52,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:52,711 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:52,711 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:52,711 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:52,711 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:52,711 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:52,711 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:52,711 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:52,711 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:52,711 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:52,711 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:52,711 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:52,712 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:52,712 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,712 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:52,712 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:52,713 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,713 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:52,714 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,714 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:52,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:52,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66da0564[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:52,715 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:52,715 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,717 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,717 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:52,717 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:52,717 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:52,717 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,717 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:52,720 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:52,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:52,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:52,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:52,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,721 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:52,721 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:52,721 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:52,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:52,723 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:52,723 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:52,723 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:52,723 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:52,723 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:52,727 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,727 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:52,727 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:52,728 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,732 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:52,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,736 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:52,736 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:52,736 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:52,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:52,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:52,736 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:52,736 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:52,740 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,740 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:52,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:52,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:52,740 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:52,740 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,740 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,740 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,740 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:52,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:52,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681844152730, max timestamp=1681844152757Num uniques 1104; 2023-04-18 18:55:52,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-18 18:55:52,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681844152730, max timestamp=1681844152757Num uniques 1104; 2023-04-18 18:55:52,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-18 18:55:52,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681844152730, max timestamp=1681844152757Num uniques 1104; 2023-04-18 18:55:52,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-18 18:55:52,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681844152783, max timestamp=1681844152797Num uniques 1113; 2023-04-18 18:55:52,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681844152783, max timestamp=1681844152797Num uniques 1113; 2023-04-18 18:55:52,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681844152783, max timestamp=1681844152797Num uniques 1113; 2023-04-18 18:55:52,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-04-18 18:55:52,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-04-18 18:55:52,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-04-18 18:55:52,836 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:52,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1681844152822, max timestamp=1681844152835Num uniques 1122; 2023-04-18 18:55:52,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=14742, suffix heap size=144656,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=322 2023-04-18 18:55:52,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1681844152822, max timestamp=1681844152835Num uniques 1122; 2023-04-18 18:55:52,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=14742, suffix heap size=144656,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=322 2023-04-18 18:55:52,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1681844152822, max timestamp=1681844152835Num uniques 1122; 2023-04-18 18:55:52,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=14742, suffix heap size=144656,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=322 2023-04-18 18:55:52,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:52,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:52,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:52,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:52,841 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:52,841 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:52,841 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:52,841 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:52,842 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,842 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:52,843 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,843 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:52,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:52,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:52,843 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,844 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:52,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:52,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:52,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:52,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:52,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:52,845 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:52,845 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:52,845 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:52,845 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:52,845 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:52,845 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:52,845 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:52,845 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:52,845 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:52,845 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:52,845 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:52,845 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:52,845 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,845 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:52,845 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:52,847 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,847 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:52,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:52,848 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,848 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:52,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:52,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:52,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e05bf85[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:52,851 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:52,851 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,852 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,853 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:52,853 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:52,853 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:52,853 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,853 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:52,854 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,854 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:52,854 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:52,854 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:52,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:52,856 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:52,856 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:52,856 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:52,856 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:52,856 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:52,861 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,861 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:52,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,861 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:52,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:52,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:52,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:52,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:52,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,862 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:52,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,864 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:52,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,868 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:52,869 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:52,869 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:52,869 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:52,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:52,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:52,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:52,869 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:52,869 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:52,869 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,869 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,869 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,869 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,869 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:52,869 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:52,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:52,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:52,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:52,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844212873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1681844152862, max timestamp=1681844152875Num uniques 1181; 2023-04-18 18:55:52,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1681844152862, max timestamp=1681844152875Num uniques 1181; 2023-04-18 18:55:52,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=12866, suffix heap size=150792,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=281 2023-04-18 18:55:52,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1681844152862, max timestamp=1681844152875Num uniques 1181; 2023-04-18 18:55:52,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=12866, suffix heap size=150792,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=281 2023-04-18 18:55:52,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=12866, suffix heap size=150792,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=281 2023-04-18 18:55:52,895 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/b253086fd2ce4405ae88d270cdb2f443 2023-04-18 18:55:52,905 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/3ee3767e85ed4e319c4d97806bfddf86 2023-04-18 18:55:52,908 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/31050cfe79564201af448e84127f29c3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/31050cfe79564201af448e84127f29c3 2023-04-18 18:55:52,914 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/31050cfe79564201af448e84127f29c3, entries=200, sequenceid=318, filesize=14.3 K 2023-04-18 18:55:52,918 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/b253086fd2ce4405ae88d270cdb2f443 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/b253086fd2ce4405ae88d270cdb2f443 2023-04-18 18:55:52,922 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/b253086fd2ce4405ae88d270cdb2f443, entries=150, sequenceid=318, filesize=11.9 K 2023-04-18 18:55:52,923 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/3ee3767e85ed4e319c4d97806bfddf86 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/3ee3767e85ed4e319c4d97806bfddf86 2023-04-18 18:55:52,937 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/3ee3767e85ed4e319c4d97806bfddf86, entries=150, sequenceid=318, filesize=11.9 K 2023-04-18 18:55:52,937 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~64.50 KB/66048, currentSize=140.89 KB/144270 for 814b3c470b4969ca1c95de968f5dec66 in 474ms, sequenceid=318, compaction requested=true 2023-04-18 18:55:52,938 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:52,938 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:52,938 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:55:52,938 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:55:52,938 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:52,938 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:52,940 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53702 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:55:52,940 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/A is initiating minor compaction (all files) 2023-04-18 18:55:52,940 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/A in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:52,940 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/9f3c3ffc0aec458da1da795caa7bcaf5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/87b30d6f0b524e469669bf38157f6bf1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/cfab6d83b19842aa8590121b3e7f5fa2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/31050cfe79564201af448e84127f29c3] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=52.4 K 2023-04-18 18:55:52,940 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48872 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:55:52,941 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/B is initiating minor compaction (all files) 2023-04-18 18:55:52,941 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/B in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:52,941 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/9ad2742c6a774191b0ef8f27a66a62a2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/f977335a15324f4e8cde60ed293163f6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/41f59f53c20d49cd8572e6e0869697d9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/b253086fd2ce4405ae88d270cdb2f443] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=47.7 K 2023-04-18 18:55:52,941 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 9f3c3ffc0aec458da1da795caa7bcaf5, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681844151327 2023-04-18 18:55:52,941 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 9ad2742c6a774191b0ef8f27a66a62a2, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681844151327 2023-04-18 18:55:52,941 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 87b30d6f0b524e469669bf38157f6bf1, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1681844151713 2023-04-18 18:55:52,941 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting f977335a15324f4e8cde60ed293163f6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1681844151713 2023-04-18 18:55:52,941 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting cfab6d83b19842aa8590121b3e7f5fa2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1681844151746 2023-04-18 18:55:52,942 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 41f59f53c20d49cd8572e6e0869697d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1681844151746 2023-04-18 18:55:52,942 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 31050cfe79564201af448e84127f29c3, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681844152173 2023-04-18 18:55:52,942 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b253086fd2ce4405ae88d270cdb2f443, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681844152173 2023-04-18 18:55:52,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681844152887, max timestamp=1681844152938Num uniques 1140; 2023-04-18 18:55:52,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-04-18 18:55:52,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:52,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681844152887, max timestamp=1681844152938Num uniques 1140; 2023-04-18 18:55:52,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-04-18 18:55:52,944 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=73.80 KB heapSize=197.05 KB 2023-04-18 18:55:52,944 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:52,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:52,944 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:52,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681844152887, max timestamp=1681844152938Num uniques 1140; 2023-04-18 18:55:52,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-04-18 18:55:52,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:52,944 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:52,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:52,960 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#A#compaction#441 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:52,961 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#B#compaction#442 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:52,969 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:52,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:52,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:52,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:52,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:52,976 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:52,976 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:52,976 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:52,976 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:52,977 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,977 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:52,978 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,978 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:52,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:52,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:52,978 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,978 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:52,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:52,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:52,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:52,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:52,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:52,985 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:52,985 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:52,985 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:52,985 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:52,985 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:52,985 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:52,985 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:52,985 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:52,985 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:52,985 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:52,985 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:52,985 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,985 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:52,985 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:52,986 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,987 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:52,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:52,987 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,987 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:52,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:52,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:52,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:52,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:52,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@537ef843[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:52,988 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:52,988 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,991 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:52,992 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:52,992 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:52,992 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:52,992 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,992 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:52,993 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,993 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:52,993 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:52,993 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:52,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:52,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:52,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:52,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:52,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:52,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:52,996 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:52,996 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:52,996 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:52,996 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:52,996 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:52,997 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,997 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:52,997 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,005 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,006 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,009 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,009 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,009 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,009 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,009 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,009 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,009 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,009 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:53,009 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:53,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:53,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:53,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:53,009 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:53,009 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:53,010 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:53,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:53,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681844152996, max timestamp=1681844153020Num uniques 859; 2023-04-18 18:55:53,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681844152996, max timestamp=1681844153020Num uniques 859; 2023-04-18 18:55:53,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 18:55:53,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 18:55:53,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681844152996, max timestamp=1681844153020Num uniques 859; 2023-04-18 18:55:53,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 18:55:53,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7682, totalHeapSize=17712, min timestamp=1681844153045, max timestamp=1681844153075Num uniques 1118; 2023-04-18 18:55:53,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51192, new segment data size=7682, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-18 18:55:53,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7680, totalHeapSize=17712, min timestamp=1681844153045, max timestamp=1681844153075Num uniques 1118; 2023-04-18 18:55:53,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51190, new segment data size=7680, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-18 18:55:53,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7682, totalHeapSize=17712, min timestamp=1681844153045, max timestamp=1681844153075Num uniques 1118; 2023-04-18 18:55:53,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51192, new segment data size=7682, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-18 18:55:53,110 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:53,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:53,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:53,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:53,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:53,116 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:53,116 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:53,116 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:53,116 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:53,117 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,117 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,119 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:53,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:53,119 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,120 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,120 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:53,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:53,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:53,124 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:53,124 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:53,124 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:53,126 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:53,126 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:53,126 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:53,127 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:53,127 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:53,127 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:53,127 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:53,127 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:53,127 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,127 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:53,127 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:53,129 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,129 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:53,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,129 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,129 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:53,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:53,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@435ef69d[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:53,134 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:53,134 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,135 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,136 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,136 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:53,136 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:53,136 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:53,136 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:53,138 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,138 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:53,138 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:53,138 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:53,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:53,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:53,141 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:53,141 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:53,141 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:53,141 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:53,141 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:53,143 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,143 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,143 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:53,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,144 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,145 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681844153130, max timestamp=1681844153146Num uniques 1027; 2023-04-18 18:55:53,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 18:55:53,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681844153130, max timestamp=1681844153146Num uniques 1027; 2023-04-18 18:55:53,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 18:55:53,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681844153130, max timestamp=1681844153146Num uniques 1027; 2023-04-18 18:55:53,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 18:55:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,152 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:53,152 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:53,152 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:53,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:53,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:53,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:53,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:53,153 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:53,153 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:53,152 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,153 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,153 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,153 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,153 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,153 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,152 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:53,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:53,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:53,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681844153183, max timestamp=1681844153195Num uniques 1036; 2023-04-18 18:55:53,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-18 18:55:53,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681844153183, max timestamp=1681844153195Num uniques 1036; 2023-04-18 18:55:53,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-18 18:55:53,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681844153183, max timestamp=1681844153195Num uniques 1036; 2023-04-18 18:55:53,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-18 18:55:53,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681844153221, max timestamp=1681844153239Num uniques 1045; 2023-04-18 18:55:53,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-18 18:55:53,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11210, totalHeapSize=25720, min timestamp=1681844153221, max timestamp=1681844153239Num uniques 1045; 2023-04-18 18:55:53,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11210, totalHeapSize=25720, min timestamp=1681844153221, max timestamp=1681844153239Num uniques 1045; 2023-04-18 18:55:53,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47850, new segment data size=11210, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-18 18:55:53,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47850, new segment data size=11210, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-18 18:55:53,253 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:53,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:53,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:53,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:53,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:53,256 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:53,256 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:53,256 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:53,256 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:53,257 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,257 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,258 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,258 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:53,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:53,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:53,259 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,259 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:53,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:53,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:53,260 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:53,260 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:53,260 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:53,260 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:53,262 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,262 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,262 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,262 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:53,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:53,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75993ab7[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:53,266 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:53,266 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,268 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,268 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:53,268 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,268 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:53,268 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:53,268 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:53,270 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:53,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,270 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:53,270 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:53,270 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:53,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:53,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:53,272 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:53,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:53,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:53,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:53,272 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:53,274 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,274 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:53,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,274 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:53,275 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,276 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,288 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,288 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,288 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,288 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,288 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,288 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,288 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,288 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:53,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:53,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:53,288 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:53,289 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:53,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681844153271, max timestamp=1681844153286Num uniques 1104; 2023-04-18 18:55:53,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-18 18:55:53,289 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:53,289 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:53,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:53,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681844153271, max timestamp=1681844153286Num uniques 1104; 2023-04-18 18:55:53,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-18 18:55:53,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:53,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:53,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681844153271, max timestamp=1681844153286Num uniques 1104; 2023-04-18 18:55:53,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-18 18:55:53,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681844153313, max timestamp=1681844153325Num uniques 1063; 2023-04-18 18:55:53,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681844153313, max timestamp=1681844153325Num uniques 1063; 2023-04-18 18:55:53,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681844153313, max timestamp=1681844153325Num uniques 1063; 2023-04-18 18:55:53,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-18 18:55:53,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-18 18:55:53,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-18 18:55:53,364 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/cf7bc210489c4cfba44bd2c6e25dd0b0 2023-04-18 18:55:53,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681844153353, max timestamp=1681844153367Num uniques 1122; 2023-04-18 18:55:53,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-18 18:55:53,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681844153353, max timestamp=1681844153367Num uniques 1122; 2023-04-18 18:55:53,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-18 18:55:53,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681844153353, max timestamp=1681844153367Num uniques 1122; 2023-04-18 18:55:53,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-04-18 18:55:53,378 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/fadd7dbc8f5042c981c525aff5a758b5 2023-04-18 18:55:53,386 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/c36ebe44abbd4270b41b8b5bfbc8503e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/c36ebe44abbd4270b41b8b5bfbc8503e 2023-04-18 18:55:53,388 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/722166b5d1a4445aa274d3b45e265dca as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/722166b5d1a4445aa274d3b45e265dca 2023-04-18 18:55:53,389 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:53,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:53,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:53,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:53,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:53,396 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:53,396 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:53,396 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:53,396 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:53,397 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,397 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,398 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:53,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:53,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:53,398 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,399 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,399 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:53,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:53,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:53,401 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/B of 814b3c470b4969ca1c95de968f5dec66 into c36ebe44abbd4270b41b8b5bfbc8503e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:53,402 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/A of 814b3c470b4969ca1c95de968f5dec66 into 722166b5d1a4445aa274d3b45e265dca(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:53,402 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:53,402 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:53,402 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/B, priority=12, startTime=1681844152938; duration=0sec 2023-04-18 18:55:53,402 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/A, priority=12, startTime=1681844152938; duration=0sec 2023-04-18 18:55:53,402 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:53,402 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:53,402 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:55:53,403 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48872 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:55:53,403 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/C is initiating minor compaction (all files) 2023-04-18 18:55:53,403 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/C in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:53,403 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f8a13fa737214ae1b84f35213bf333e3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/47238a0f2a554909bcd2aa62bdb4d153, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f67cf587c48b493c826cb6548975ab92, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/3ee3767e85ed4e319c4d97806bfddf86] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=47.7 K 2023-04-18 18:55:53,404 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting f8a13fa737214ae1b84f35213bf333e3, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681844151327 2023-04-18 18:55:53,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:53,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:53,404 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:53,404 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:53,404 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 47238a0f2a554909bcd2aa62bdb4d153, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1681844151713 2023-04-18 18:55:53,404 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:53,404 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:53,404 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:53,404 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting f67cf587c48b493c826cb6548975ab92, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1681844151746 2023-04-18 18:55:53,405 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:53,405 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:53,405 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 3ee3767e85ed4e319c4d97806bfddf86, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681844152173 2023-04-18 18:55:53,405 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:53,405 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:53,405 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:53,405 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:53,405 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,405 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:53,405 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:53,407 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,407 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,407 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,407 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:53,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16834a16[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:53,409 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:53,409 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,410 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,410 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:53,411 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:53,411 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:53,411 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,411 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:53,416 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:53,416 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:53,416 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:53,417 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:53,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1681844153389, max timestamp=1681844153416Num uniques 1081; 2023-04-18 18:55:53,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=15156, suffix heap size=140392,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=331 2023-04-18 18:55:53,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1681844153389, max timestamp=1681844153416Num uniques 1081; 2023-04-18 18:55:53,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=15156, suffix heap size=140392,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=331 2023-04-18 18:55:53,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1681844153389, max timestamp=1681844153416Num uniques 1031; 2023-04-18 18:55:53,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=15156, suffix heap size=135192,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=331 2023-04-18 18:55:53,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:53,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:53,425 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:53,425 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:53,425 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:53,425 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:53,425 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:53,426 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,426 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,426 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:53,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:53,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:53,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:53,427 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#C#compaction#445 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:53,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,429 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,430 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,434 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:53,435 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,435 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:53,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:53,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:53,435 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:53,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:53,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:53,435 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:53,435 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,435 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:53,435 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,435 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,435 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,435 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,435 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:53,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:53,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:53,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1681844153444, max timestamp=1681844153456Num uniques 1190; 2023-04-18 18:55:53,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=13280, suffix heap size=151728,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=290 2023-04-18 18:55:53,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681844153444, max timestamp=1681844153456Num uniques 1190; 2023-04-18 18:55:53,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=10990, suffix heap size=151728,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=240 2023-04-18 18:55:53,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681844153444, max timestamp=1681844153461Num uniques 1190; 2023-04-18 18:55:53,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=8700, suffix heap size=151728,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=190 2023-04-18 18:55:53,464 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/b9179739a8a84945b53eb6e5b8aa3092 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/b9179739a8a84945b53eb6e5b8aa3092 2023-04-18 18:55:53,468 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/C of 814b3c470b4969ca1c95de968f5dec66 into b9179739a8a84945b53eb6e5b8aa3092(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:53,468 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:53,468 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/C, priority=12, startTime=1681844152938; duration=0sec 2023-04-18 18:55:53,468 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:53,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:53,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844213488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:53,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844213493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11394, totalHeapSize=26136, min timestamp=1681844153485, max timestamp=1681844153499Num uniques 1149; 2023-04-18 18:55:53,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52614, new segment data size=11394, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-18 18:55:53,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11394, totalHeapSize=26136, min timestamp=1681844153485, max timestamp=1681844153499Num uniques 1049; 2023-04-18 18:55:53,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48034, new segment data size=11394, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-18 18:55:53,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11394, totalHeapSize=26136, min timestamp=1681844153485, max timestamp=1681844153499Num uniques 1099; 2023-04-18 18:55:53,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50324, new segment data size=11394, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-18 18:55:53,535 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:53,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:53,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:53,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:53,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:53,538 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:53,538 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:53,538 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:53,538 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:53,539 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,539 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,540 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:53,540 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:53,540 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,540 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:53,541 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:53,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:53,542 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:53,542 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:53,542 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:53,542 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:53,542 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:53,542 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:53,542 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:53,542 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:53,542 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:53,542 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:53,542 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:53,542 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,542 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:53,542 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:53,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681844153525, max timestamp=1681844153543Num uniques 1108; 2023-04-18 18:55:53,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=11810, suffix heap size=143200,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=258 2023-04-18 18:55:53,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681844153525, max timestamp=1681844153543Num uniques 1108; 2023-04-18 18:55:53,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=11810, suffix heap size=143200,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=258 2023-04-18 18:55:53,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681844153525, max timestamp=1681844153543Num uniques 1108; 2023-04-18 18:55:53,547 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=11810, suffix heap size=143200,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=258 2023-04-18 18:55:53,547 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:53,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,548 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,548 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:53,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@329b1465[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:53,549 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:53,549 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,550 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,550 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:53,550 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:53,550 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:53,550 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,550 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:53,552 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,552 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:53,552 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:53,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:53,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,553 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:53,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:53,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:53,556 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:53,556 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:53,556 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:53,557 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:53,557 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:53,558 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,558 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,558 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:53,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,560 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,561 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,566 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:53,566 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:53,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:53,566 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:53,566 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:53,566 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:53,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:53,566 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,566 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,566 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,566 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,566 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,566 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:53,567 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:53,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:53,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681844153563, max timestamp=1681844153585Num uniques 1117; 2023-04-18 18:55:53,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=7642, suffix heap size=144136,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=167 2023-04-18 18:55:53,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681844153563, max timestamp=1681844153585Num uniques 1117; 2023-04-18 18:55:53,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=7642, suffix heap size=144136,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=167 2023-04-18 18:55:53,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681844153563, max timestamp=1681844153585Num uniques 1117; 2023-04-18 18:55:53,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=7642, suffix heap size=144136,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=167 2023-04-18 18:55:53,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1681844153608, max timestamp=1681844153617Num uniques 1026; 2023-04-18 18:55:53,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=14926, suffix heap size=134672,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=326 2023-04-18 18:55:53,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1681844153608, max timestamp=1681844153617Num uniques 1026; 2023-04-18 18:55:53,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=14926, suffix heap size=134672,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=326 2023-04-18 18:55:53,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1681844153608, max timestamp=1681844153617Num uniques 1026; 2023-04-18 18:55:53,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=14926, suffix heap size=134672,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=326 2023-04-18 18:55:53,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:53,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844213656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1681844153648, max timestamp=1681844153662Num uniques 1185; 2023-04-18 18:55:53,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=15340, suffix heap size=151208,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=335 2023-04-18 18:55:53,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:53,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844213658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1681844153648, max timestamp=1681844153662Num uniques 1185; 2023-04-18 18:55:53,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=15340, suffix heap size=151208,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=335 2023-04-18 18:55:53,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1681844153648, max timestamp=1681844153662Num uniques 1185; 2023-04-18 18:55:53,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=15340, suffix heap size=151208,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=335 2023-04-18 18:55:53,667 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:53,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:53,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:53,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:53,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:53,670 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:53,670 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:53,670 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:53,670 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:53,670 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,670 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,671 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,671 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:53,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,673 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,673 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:53,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:53,673 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:53,673 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:53,673 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:53,676 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:53,676 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:53,676 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:53,676 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:53,678 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,678 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:53,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,678 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,678 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:53,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:53,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@645f6b5[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:53,679 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:53,679 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,680 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,681 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:53,681 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:53,681 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:53,681 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,681 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:53,683 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,683 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,683 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:53,683 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:53,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:53,686 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:53,686 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:53,686 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:53,686 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:53,686 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:53,688 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,688 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,688 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:53,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,689 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,692 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,698 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,698 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:53,698 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:53,698 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:53,698 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:53,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:53,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:53,698 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,698 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,698 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,698 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,698 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,698 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:53,699 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:53,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:53,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1681844153698, max timestamp=1681844153747Num uniques 1194; 2023-04-18 18:55:53,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=13464, suffix heap size=152144,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=294 2023-04-18 18:55:53,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1681844153698, max timestamp=1681844153747Num uniques 1194; 2023-04-18 18:55:53,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=13464, suffix heap size=152144,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=294 2023-04-18 18:55:53,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1681844153698, max timestamp=1681844153747Num uniques 1194; 2023-04-18 18:55:53,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=13464, suffix heap size=152144,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=294 2023-04-18 18:55:53,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9288, totalHeapSize=21352, min timestamp=1681844153762, max timestamp=1681844153789Num uniques 1153; 2023-04-18 18:55:53,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9288, totalHeapSize=21352, min timestamp=1681844153762, max timestamp=1681844153789Num uniques 1153; 2023-04-18 18:55:53,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52798, new segment data size=9288, suffix heap size=147880,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=203 2023-04-18 18:55:53,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52798, new segment data size=9288, suffix heap size=147880,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=203 2023-04-18 18:55:53,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1681844153762, max timestamp=1681844153789Num uniques 1153; 2023-04-18 18:55:53,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=9295, suffix heap size=147880,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=203 2023-04-18 18:55:53,799 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:53,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:53,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:53,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:53,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:53,803 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:53,803 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:53,803 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:53,803 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:53,804 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,804 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,808 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:53,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:53,808 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,808 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,809 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:53,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:53,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:53,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:53,809 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:53,810 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:53,810 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:53,810 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:53,810 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:53,810 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:53,810 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:53,810 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:53,810 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:53,810 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:53,810 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:53,810 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,810 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:53,810 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:53,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:53,814 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:53,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:53,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a352848[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:53,817 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:53,817 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,818 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,818 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:53,818 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:53,818 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:53,818 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,818 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:53,820 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,820 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:53,820 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,820 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:53,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:53,823 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:53,823 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:53,823 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:53,823 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:53,823 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:53,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=362, cellSize=16572, totalHeapSize=37888, min timestamp=1681844153791, max timestamp=1681844153824Num uniques 1062; 2023-04-18 18:55:53,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=362, cellSize=16572, totalHeapSize=37888, min timestamp=1681844153791, max timestamp=1681844153824Num uniques 1062; 2023-04-18 18:55:53,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=16572, suffix heap size=138416,new segment heap size=37888 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=362 2023-04-18 18:55:53,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=16572, suffix heap size=138416,new segment heap size=37888 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=362 2023-04-18 18:55:53,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681844153815, max timestamp=1681844153824Num uniques 1062; 2023-04-18 18:55:53,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=6870, suffix heap size=138416,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=150 2023-04-18 18:55:53,830 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,830 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:53,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,830 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:53,831 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/adc459e664974b4db62421bef63671b9 2023-04-18 18:55:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,832 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,834 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:53,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:53,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:53,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:53,837 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/cf7bc210489c4cfba44bd2c6e25dd0b0 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/cf7bc210489c4cfba44bd2c6e25dd0b0 2023-04-18 18:55:53,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:53,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:53,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,838 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:53,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:53,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,838 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:53,838 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:53,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:53,841 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/cf7bc210489c4cfba44bd2c6e25dd0b0, entries=250, sequenceid=493, filesize=16.7 K 2023-04-18 18:55:53,842 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/fadd7dbc8f5042c981c525aff5a758b5 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/fadd7dbc8f5042c981c525aff5a758b5 2023-04-18 18:55:53,856 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/fadd7dbc8f5042c981c525aff5a758b5, entries=150, sequenceid=493, filesize=11.9 K 2023-04-18 18:55:53,857 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/adc459e664974b4db62421bef63671b9 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/adc459e664974b4db62421bef63671b9 2023-04-18 18:55:53,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1681844153843, max timestamp=1681844153857Num uniques 1221; 2023-04-18 18:55:53,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1681844153843, max timestamp=1681844153857Num uniques 1221; 2023-04-18 18:55:53,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=55916, new segment data size=12406, suffix heap size=154952,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1221, new segment cells count=271 2023-04-18 18:55:53,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=55916, new segment data size=12406, suffix heap size=154952,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1221, new segment cells count=271 2023-04-18 18:55:53,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1681844153843, max timestamp=1681844153857Num uniques 1009; 2023-04-18 18:55:53,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=12406, suffix heap size=132904,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=271 2023-04-18 18:55:53,870 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/adc459e664974b4db62421bef63671b9, entries=150, sequenceid=493, filesize=11.9 K 2023-04-18 18:55:53,871 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~78.52 KB/80400, currentSize=80.51 KB/82440 for 814b3c470b4969ca1c95de968f5dec66 in 927ms, sequenceid=493, compaction requested=false 2023-04-18 18:55:53,871 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:53,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:53,872 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=87.22 KB heapSize=235.03 KB 2023-04-18 18:55:53,872 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:53,872 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:53,872 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:53,872 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:53,872 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:53,872 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:53,889 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=833 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/7c3e1fe04c174fa4a640f586153bbf59 2023-04-18 18:55:53,922 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=833 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/8c04539bdbc0461cb31df6fd1bfc2e9c 2023-04-18 18:55:53,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:53,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844213910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:53,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844213911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:53,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844213912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:53,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844213912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:53,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844213913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,938 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:53,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:53,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:53,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:53,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:53,947 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:53,947 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:53,947 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:53,947 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:53,949 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,949 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,950 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,950 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:53,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:53,951 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,951 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:53,952 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=833 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/db9638f1c4ad42c796f474ded5df240b 2023-04-18 18:55:53,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:53,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:53,952 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:53,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:53,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:53,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:53,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:53,953 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:53,953 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:53,953 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:53,953 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:53,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:53,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:53,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:53,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:53,955 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,955 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,955 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,955 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,956 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/7c3e1fe04c174fa4a640f586153bbf59 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/7c3e1fe04c174fa4a640f586153bbf59 2023-04-18 18:55:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36ea16e2[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:53,957 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:53,958 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,960 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,960 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/7c3e1fe04c174fa4a640f586153bbf59, entries=300, sequenceid=833, filesize=19.0 K 2023-04-18 18:55:53,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,960 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:53,960 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:53,960 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:53,961 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/8c04539bdbc0461cb31df6fd1bfc2e9c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8c04539bdbc0461cb31df6fd1bfc2e9c 2023-04-18 18:55:53,962 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,962 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:53,963 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,963 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:53,963 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:53,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:53,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,963 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:53,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:53,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:53,965 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:53,965 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:53,965 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:53,965 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:53,965 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:53,967 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,967 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,967 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:53,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:53,967 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:53,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:53,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:53,969 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,970 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8c04539bdbc0461cb31df6fd1bfc2e9c, entries=150, sequenceid=833, filesize=11.9 K 2023-04-18 18:55:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,971 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/db9638f1c4ad42c796f474ded5df240b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/db9638f1c4ad42c796f474ded5df240b 2023-04-18 18:55:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:53,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:53,973 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,973 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,973 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:53,973 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:53,973 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,973 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:53,973 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:53,973 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:53,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:53,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:53,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:53,973 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:53,973 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:53,973 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:53,973 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:53,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:53,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:53,975 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/db9638f1c4ad42c796f474ded5df240b, entries=150, sequenceid=833, filesize=11.9 K 2023-04-18 18:55:53,976 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~234.98 KB/240624, currentSize=93.93 KB/96180 for 814b3c470b4969ca1c95de968f5dec66 in 104ms, sequenceid=833, compaction requested=true 2023-04-18 18:55:53,976 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:53,976 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:53,976 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:53,976 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:53,976 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:53,976 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:53,981 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 49372 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:53,981 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:53,981 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/A is initiating minor compaction (all files) 2023-04-18 18:55:53,981 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/B is initiating minor compaction (all files) 2023-04-18 18:55:53,982 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/A in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:53,982 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/B in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:53,982 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/722166b5d1a4445aa274d3b45e265dca, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/cf7bc210489c4cfba44bd2c6e25dd0b0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/7c3e1fe04c174fa4a640f586153bbf59] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=48.2 K 2023-04-18 18:55:53,982 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/c36ebe44abbd4270b41b8b5bfbc8503e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/fadd7dbc8f5042c981c525aff5a758b5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8c04539bdbc0461cb31df6fd1bfc2e9c] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=36.3 K 2023-04-18 18:55:53,982 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 722166b5d1a4445aa274d3b45e265dca, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681844152173 2023-04-18 18:55:53,982 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting c36ebe44abbd4270b41b8b5bfbc8503e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681844152173 2023-04-18 18:55:53,984 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting fadd7dbc8f5042c981c525aff5a758b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1681844152920 2023-04-18 18:55:53,984 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting cf7bc210489c4cfba44bd2c6e25dd0b0, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1681844152887 2023-04-18 18:55:53,984 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 8c04539bdbc0461cb31df6fd1bfc2e9c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=833, earliestPutTs=1681844153843 2023-04-18 18:55:53,984 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 7c3e1fe04c174fa4a640f586153bbf59, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=833, earliestPutTs=1681844153843 2023-04-18 18:55:53,993 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#B#compaction#449 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:53,993 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#A#compaction#450 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:54,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:54,027 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-18 18:55:54,027 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:54,027 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/c5254c10f7044f5c90d93e197a28de81 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/c5254c10f7044f5c90d93e197a28de81 2023-04-18 18:55:54,027 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:54,027 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:54,027 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:54,028 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:54,028 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:54,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:54,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844214036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,047 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/26d5eac8fa074e59b6e787df250caa1e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/26d5eac8fa074e59b6e787df250caa1e 2023-04-18 18:55:54,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:54,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844214037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,048 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/B of 814b3c470b4969ca1c95de968f5dec66 into c5254c10f7044f5c90d93e197a28de81(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:54,048 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:54,048 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/B, priority=13, startTime=1681844153976; duration=0sec 2023-04-18 18:55:54,048 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:54,048 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:54,049 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:54,049 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/C is initiating minor compaction (all files) 2023-04-18 18:55:54,050 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/C in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:54,050 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/b9179739a8a84945b53eb6e5b8aa3092, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/adc459e664974b4db62421bef63671b9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/db9638f1c4ad42c796f474ded5df240b] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=36.3 K 2023-04-18 18:55:54,050 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b9179739a8a84945b53eb6e5b8aa3092, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681844152173 2023-04-18 18:55:54,050 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting adc459e664974b4db62421bef63671b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1681844152920 2023-04-18 18:55:54,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:54,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844214043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,051 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting db9638f1c4ad42c796f474ded5df240b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=833, earliestPutTs=1681844153843 2023-04-18 18:55:54,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:54,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844214044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:54,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844214045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,055 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/A of 814b3c470b4969ca1c95de968f5dec66 into 26d5eac8fa074e59b6e787df250caa1e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:54,055 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:54,055 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/A, priority=13, startTime=1681844153976; duration=0sec 2023-04-18 18:55:54,056 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:54,061 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#C#compaction#452 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:54,073 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:54,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:54,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:54,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:54,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:54,080 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:54,080 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:54,080 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:54,080 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:54,080 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,081 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,082 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:54,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:54,082 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:54,083 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,083 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:54,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:54,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:54,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:54,088 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:54,088 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:54,088 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:54,095 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:54,095 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:54,095 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:54,095 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:54,096 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:54,096 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:54,096 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:54,096 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:54,096 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,096 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:54,096 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:54,099 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,099 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:54,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:54,099 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,099 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:54,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:54,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:54,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:54,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:54,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cb80466[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:54,100 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:54,101 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,102 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,102 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:54,102 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:54,102 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:54,102 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,102 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:54,104 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,105 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:54,105 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:54,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:54,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:54,105 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:54,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:54,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:54,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:54,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:54,109 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:54,110 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:54,110 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:54,110 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:54,110 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:54,112 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,112 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,112 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:54,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:54,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,113 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,114 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:54,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,120 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,120 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,120 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,120 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,120 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,120 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,120 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,120 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:54,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:54,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:54,121 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:54,121 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:54,121 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:54,121 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:54,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:54,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:54,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:54,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844214146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:54,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844214148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:54,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844214152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:54,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844214155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:54,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844214156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,221 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:54,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:54,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:54,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:54,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:54,227 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:54,227 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:54,227 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:54,227 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:54,227 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,227 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,228 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:54,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:54,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:54,228 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,229 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,229 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:54,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:54,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:54,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:54,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:54,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:54,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:54,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:54,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:54,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:54,232 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:54,232 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:54,232 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:54,232 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:54,233 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:54,233 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:54,233 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,233 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:54,233 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:54,239 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,239 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:54,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:54,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75e1b11a[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:54,243 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:54,243 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,244 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:54,244 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,244 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:54,246 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:54,246 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:54,246 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:54,246 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:54,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:54,250 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:54,251 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:54,251 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:54,251 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:54,251 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:54,252 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:54,253 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,253 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:54,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:54,254 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,257 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,264 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,264 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,264 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,264 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:54,264 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:54,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:54,264 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:54,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:54,264 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:54,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:54,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:54,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:54,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:54,265 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:54,265 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:54,265 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,265 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,265 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:54,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844214350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:54,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844214353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:54,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844214356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,365 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:54,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:54,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:54,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:54,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844214360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:54,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844214361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:54,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:54,368 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:54,368 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:54,368 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:54,368 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:54,373 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,373 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,374 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:54,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:54,374 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,375 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,375 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:54,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:54,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:54,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:54,380 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:54,380 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:54,380 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:54,387 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:54,387 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:54,387 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:54,387 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:54,388 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:54,388 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:54,390 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:54,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:54,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:54,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:54,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:54,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6685a086[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:54,392 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:54,393 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,394 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,394 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:54,394 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:54,394 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:54,395 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,395 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:54,396 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,396 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:54,396 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:54,397 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:54,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:54,399 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:54,399 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:54,399 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:54,399 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:54,399 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:54,401 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,401 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:54,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,401 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:54,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,401 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,403 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,406 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:54,406 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:54,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:54,406 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,406 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:54,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:54,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:54,406 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,406 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,406 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,406 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,406 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,406 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,407 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:54,407 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:54,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:54,464 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=851 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/372b2d4df0434946a45bf31f8fe38ff9 2023-04-18 18:55:54,481 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/fdf754176e1842be887864ec041b8566 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/fdf754176e1842be887864ec041b8566 2023-04-18 18:55:54,485 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/C of 814b3c470b4969ca1c95de968f5dec66 into fdf754176e1842be887864ec041b8566(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:54,485 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:54,485 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/C, priority=13, startTime=1681844153976; duration=0sec 2023-04-18 18:55:54,485 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:54,507 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:54,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:54,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:54,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:54,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:54,510 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:54,510 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:54,511 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:54,511 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:54,511 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,511 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,512 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:54,512 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:54,512 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,512 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:54,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:54,514 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:54,514 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:54,514 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:54,514 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:54,514 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:54,517 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,517 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:54,517 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,517 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:54,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:54,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ef7b2[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:54,519 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:54,520 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,521 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,521 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:54,521 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:54,521 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:54,521 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,521 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:54,522 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:54,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:54,526 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:54,526 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:54,526 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:54,526 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:54,526 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:54,527 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,527 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,528 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:54,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:54,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:54,528 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,530 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,536 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:54,537 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:54,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:54,537 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:54,537 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:54,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:54,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:54,537 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:54,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:54,537 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,537 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,537 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,537 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,537 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,537 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:54,537 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:54,637 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:54,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:54,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:54,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:54,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:54,643 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:54,643 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:54,643 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:54,643 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:54,645 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,645 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,646 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,646 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:54,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:54,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:54,647 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,647 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:54,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:54,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:54,648 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:54,648 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:54,648 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:54,648 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:54,650 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,650 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:54,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:54,650 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:54,650 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:54,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:54,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:54,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:54,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cb948e7[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:54,652 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:54,652 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,653 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,653 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:54,653 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:54,653 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:54,653 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,653 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:54,658 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:54,658 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:54,658 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,658 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:54,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844214661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:54,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844214661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:54,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844214665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:54,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:54,667 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:54,667 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:54,667 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:54,667 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:54,667 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:54,669 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,669 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:54,669 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:54,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,670 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,671 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,673 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,673 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,673 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,673 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,673 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:54,673 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:54,673 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,673 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,673 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:54,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:54,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:54,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:54,673 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:54,674 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:54,674 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:54,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:54,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:54,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844214671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:54,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844214674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,774 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:54,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:54,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:54,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:54,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:54,777 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:54,777 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:54,777 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:54,777 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:54,778 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,778 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,779 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:54,779 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:54,779 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,780 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:54,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:54,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:54,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:54,781 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:54,781 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:54,781 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:54,782 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:54,782 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:54,782 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:54,782 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:54,782 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:54,782 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:54,782 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:54,782 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:54,782 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,782 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:54,782 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:54,785 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,785 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,785 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:54,786 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:54,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:54,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64a8530b[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:54,788 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:54,788 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,789 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,789 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:54,789 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,789 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:54,789 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:54,789 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:54,790 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:54,791 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:54,791 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:54,791 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:54,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:54,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:54,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:54,793 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:54,793 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:54,793 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:54,793 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:54,793 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:54,795 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,795 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:54,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,795 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:54,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,796 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,797 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:54,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,801 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,801 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,801 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,801 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,801 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:54,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:54,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:54,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:54,801 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,802 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:54,802 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,802 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:54,802 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:54,802 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:54,803 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:54,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:54,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:54,875 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=851 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/98f05764e9834fc091d29eb762698cba 2023-04-18 18:55:54,886 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=851 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/f7b60a88bda5418d8e9570881a0835d5 2023-04-18 18:55:54,890 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/372b2d4df0434946a45bf31f8fe38ff9 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/372b2d4df0434946a45bf31f8fe38ff9 2023-04-18 18:55:54,893 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/372b2d4df0434946a45bf31f8fe38ff9, entries=200, sequenceid=851, filesize=14.3 K 2023-04-18 18:55:54,894 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/98f05764e9834fc091d29eb762698cba as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/98f05764e9834fc091d29eb762698cba 2023-04-18 18:55:54,899 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/98f05764e9834fc091d29eb762698cba, entries=150, sequenceid=851, filesize=11.9 K 2023-04-18 18:55:54,899 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/f7b60a88bda5418d8e9570881a0835d5 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f7b60a88bda5418d8e9570881a0835d5 2023-04-18 18:55:54,902 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:54,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:54,903 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f7b60a88bda5418d8e9570881a0835d5, entries=150, sequenceid=851, filesize=11.9 K 2023-04-18 18:55:54,904 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 814b3c470b4969ca1c95de968f5dec66 in 877ms, sequenceid=851, compaction requested=false 2023-04-18 18:55:54,904 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:54,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:54,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:54,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:54,906 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:54,906 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:54,906 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:54,906 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:54,907 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,907 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,908 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:54,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:54,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:54,908 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,909 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:54,909 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:54,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:54,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:54,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:54,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:54,910 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:54,910 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:54,910 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:54,910 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:54,910 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:54,910 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:54,910 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:54,910 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-18 18:55:54,911 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:54,911 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:54,911 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:54,911 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:54,911 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:54,911 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:55,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:55,173 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:55,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844215191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844215192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844215196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844215197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844215198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,307 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844215298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,307 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844215301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844215306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844215307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844215307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,326 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=868 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/db53449058b24fd38cfc70e1abe638b2 2023-04-18 18:55:55,351 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=868 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/e1134d06c3ff4fd39e1701b85d472645 2023-04-18 18:55:55,362 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=868 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/bb0354d8e09c48889501d4442d7be77f 2023-04-18 18:55:55,366 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/db53449058b24fd38cfc70e1abe638b2 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/db53449058b24fd38cfc70e1abe638b2 2023-04-18 18:55:55,370 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/db53449058b24fd38cfc70e1abe638b2, entries=150, sequenceid=868, filesize=11.9 K 2023-04-18 18:55:55,371 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/e1134d06c3ff4fd39e1701b85d472645 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/e1134d06c3ff4fd39e1701b85d472645 2023-04-18 18:55:55,374 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/e1134d06c3ff4fd39e1701b85d472645, entries=150, sequenceid=868, filesize=11.9 K 2023-04-18 18:55:55,376 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/bb0354d8e09c48889501d4442d7be77f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/bb0354d8e09c48889501d4442d7be77f 2023-04-18 18:55:55,378 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/bb0354d8e09c48889501d4442d7be77f, entries=150, sequenceid=868, filesize=11.9 K 2023-04-18 18:55:55,379 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 814b3c470b4969ca1c95de968f5dec66 in 469ms, sequenceid=868, compaction requested=true 2023-04-18 18:55:55,379 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:55,379 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:55,379 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:55,379 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:55,379 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,380 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:55,380 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:55,382 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,382 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:55,383 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,383 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:55,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:55,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:55,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:55,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:55,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f625e88[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:55,384 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:55,384 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,385 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,385 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:55,385 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:55,385 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:55,386 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,386 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:55,388 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:55,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:55,389 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:55,389 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:55,389 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:55,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:55,391 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:55,391 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:55,391 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:55,391 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:55,391 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:55,392 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,392 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:55,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,393 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,392 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:55,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,394 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,397 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:55,397 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:55,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:55,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:55,397 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:55,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:55,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:55,397 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,398 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,398 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,398 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,398 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,398 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:55,398 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:55,398 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,398 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:55,498 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:55,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:55,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:55,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:55,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:55,502 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:55,502 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:55,502 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:55,502 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:55,503 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,503 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,504 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:55,504 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:55,504 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,504 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:55,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:55,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:55,506 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:55,506 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:55,506 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-18 18:55:55,507 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:55,507 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:55,507 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:55,507 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:55,507 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:55,507 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:55,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:55,513 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:55,518 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=887 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/a11eb6bfe0904a67a8c6ffe851efc28f 2023-04-18 18:55:55,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844215520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844215524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,529 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=887 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/2c1a34c06ec24e6297bef1bec0daa3fc 2023-04-18 18:55:55,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844215525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844215526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844215528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,539 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=887 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/6bae45fc63cf405594aafaf79499ffa3 2023-04-18 18:55:55,543 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/a11eb6bfe0904a67a8c6ffe851efc28f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/a11eb6bfe0904a67a8c6ffe851efc28f 2023-04-18 18:55:55,546 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/a11eb6bfe0904a67a8c6ffe851efc28f, entries=150, sequenceid=887, filesize=11.9 K 2023-04-18 18:55:55,547 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/2c1a34c06ec24e6297bef1bec0daa3fc as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/2c1a34c06ec24e6297bef1bec0daa3fc 2023-04-18 18:55:55,551 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/2c1a34c06ec24e6297bef1bec0daa3fc, entries=150, sequenceid=887, filesize=11.9 K 2023-04-18 18:55:55,552 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/6bae45fc63cf405594aafaf79499ffa3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/6bae45fc63cf405594aafaf79499ffa3 2023-04-18 18:55:55,555 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/6bae45fc63cf405594aafaf79499ffa3, entries=150, sequenceid=887, filesize=11.9 K 2023-04-18 18:55:55,556 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 814b3c470b4969ca1c95de968f5dec66 in 50ms, sequenceid=887, compaction requested=true 2023-04-18 18:55:55,556 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:55,556 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:55,556 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:55,556 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:55,556 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,556 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:55,557 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:55,559 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:55,559 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,559 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,559 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:55,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:55,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5896afb1[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:55,561 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:55,561 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,563 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,563 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:55,564 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,564 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:55,564 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:55,564 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:55,565 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:55,565 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:55,565 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,565 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:55,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:55,572 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:55,572 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:55,572 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:55,572 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:55,572 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:55,577 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,577 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:55,577 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:55,578 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,580 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,582 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,582 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:55,582 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:55,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:55,582 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:55,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:55,582 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:55,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:55,583 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,583 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,583 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:55,583 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:55,583 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,583 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:55,583 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:55,634 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-18 18:55:55,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:55,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:55,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:55,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:55,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:55,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:55,650 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=902 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/433f2d2cf69e477d868105bd1903fae1 2023-04-18 18:55:55,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844215665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844215666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844215666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844215667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844215668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,683 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:55,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:55,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:55,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:55,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:55,686 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:55,686 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:55,686 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:55,686 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:55,687 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,687 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,688 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:55,688 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,688 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,688 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:55,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:55,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:55,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:55,689 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:55,689 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:55,689 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:55,690 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:55,690 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:55,690 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:55,690 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:55,690 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:55,690 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:55,690 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:55,690 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:55,690 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,690 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:55,690 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:55,691 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,691 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:55,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:55,692 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,692 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:55,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:55,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:55,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:55,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e3e3a34[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:55,693 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:55,693 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,697 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,697 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:55,697 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:55,697 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:55,697 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,697 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:55,698 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:55,698 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:55,698 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:55,698 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:55,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:55,700 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:55,701 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:55,701 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:55,701 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:55,701 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:55,702 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,702 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,702 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:55,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:55,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,703 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,704 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,707 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,707 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,707 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:55,707 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:55,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:55,708 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,708 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:55,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:55,708 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,708 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:55,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:55,708 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:55,708 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:55,709 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:55,708 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844215780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844215780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844215780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844215780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844215780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,809 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:55,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:55,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:55,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:55,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:55,812 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:55,812 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:55,812 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:55,812 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:55,812 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,812 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,815 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:55,815 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:55,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:55,815 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,815 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:55,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:55,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:55,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:55,816 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:55,817 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:55,817 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:55,817 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:55,817 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:55,817 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:55,817 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:55,817 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:55,817 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:55,817 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:55,817 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:55,817 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,817 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:55,817 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:55,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,819 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:55,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:55,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:55,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:55,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28a810f3[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:55,822 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:55,822 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,823 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,823 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:55,823 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:55,823 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,823 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:55,823 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:55,824 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,824 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:55,825 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:55,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:55,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:55,825 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:55,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:55,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:55,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:55,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:55,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:55,827 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:55,827 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:55,827 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:55,827 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:55,827 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:55,831 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,831 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,831 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:55,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:55,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:55,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,832 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,833 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,836 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:55,836 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:55,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:55,836 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:55,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:55,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:55,837 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:55,837 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:55,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:55,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:55,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:55,836 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,937 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:55,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:55,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:55,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:55,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:55,940 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:55,940 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:55,940 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:55,940 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:55,941 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,941 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,942 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:55,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:55,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:55,942 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,943 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,943 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:55,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:55,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,943 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:55,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:55,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:55,944 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:55,944 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:55,944 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:55,944 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:55,944 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:55,944 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:55,944 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:55,944 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:55,944 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:55,944 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:55,944 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:55,944 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,944 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:55,944 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:55,946 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,946 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:55,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:55,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:55,946 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,946 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:55,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:55,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:55,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:55,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15839d70[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:55,947 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:55,948 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,949 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,950 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:55,950 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,950 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:55,950 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:55,950 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:55,952 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,952 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:55,952 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:55,952 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:55,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:55,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:55,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:55,954 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:55,954 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:55,954 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:55,954 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:55,954 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:55,956 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,956 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,956 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:55,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:55,956 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,958 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,963 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,963 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,963 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:55,963 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:55,963 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:55,964 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:55,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:55,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:55,964 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:55,964 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:55,964 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:55,964 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:55,964 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:55,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:55,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:55,964 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:55,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:55,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:55,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844215992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844215992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844215993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844215993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844215993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,061 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=902 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/40af8a0829224828ad0b2c21c9d48dcd 2023-04-18 18:55:56,064 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:56,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:56,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:56,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:56,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:56,069 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:56,069 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:56,069 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:56,069 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:56,069 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:56,069 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:56,071 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:56,071 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:56,071 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,071 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:56,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:56,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:56,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:56,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:56,072 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:56,072 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:56,072 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:56,073 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:56,073 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:56,073 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:56,073 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:56,073 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:56,074 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:56,074 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:56,074 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:56,074 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,074 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:56,074 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:56,074 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=902 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/579f859a4748401db218b7567108b53a 2023-04-18 18:55:56,075 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:56,076 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:56,076 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:56,076 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:56,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:56,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:56,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:56,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78054356[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:56,078 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:56,078 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:56,078 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/433f2d2cf69e477d868105bd1903fae1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/433f2d2cf69e477d868105bd1903fae1 2023-04-18 18:55:56,080 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:56,081 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,081 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:56,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:56,081 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:56,082 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:56,082 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:56,082 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/433f2d2cf69e477d868105bd1903fae1, entries=250, sequenceid=902, filesize=16.7 K 2023-04-18 18:55:56,082 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/40af8a0829224828ad0b2c21c9d48dcd as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/40af8a0829224828ad0b2c21c9d48dcd 2023-04-18 18:55:56,083 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:56,083 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:56,083 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:56,083 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:56,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:56,086 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:56,086 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:56,086 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:56,086 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:56,086 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:56,086 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/40af8a0829224828ad0b2c21c9d48dcd, entries=150, sequenceid=902, filesize=11.9 K 2023-04-18 18:55:56,087 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/579f859a4748401db218b7567108b53a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/579f859a4748401db218b7567108b53a 2023-04-18 18:55:56,088 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:56,088 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:56,088 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:56,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:56,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,090 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,090 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/579f859a4748401db218b7567108b53a, entries=150, sequenceid=902, filesize=11.9 K 2023-04-18 18:55:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:56,091 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 814b3c470b4969ca1c95de968f5dec66 in 457ms, sequenceid=902, compaction requested=true 2023-04-18 18:55:56,091 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:56,091 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:55:56,091 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:56,091 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,091 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:55:56,091 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-18 18:55:56,092 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,093 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 68966 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:55:56,093 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61646 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:55:56,093 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/A is initiating minor compaction (all files) 2023-04-18 18:55:56,093 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/B is initiating minor compaction (all files) 2023-04-18 18:55:56,093 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/A in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:56,093 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/B in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:56,093 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/26d5eac8fa074e59b6e787df250caa1e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/372b2d4df0434946a45bf31f8fe38ff9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/db53449058b24fd38cfc70e1abe638b2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/a11eb6bfe0904a67a8c6ffe851efc28f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/433f2d2cf69e477d868105bd1903fae1] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=67.3 K 2023-04-18 18:55:56,093 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/c5254c10f7044f5c90d93e197a28de81, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/98f05764e9834fc091d29eb762698cba, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/e1134d06c3ff4fd39e1701b85d472645, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/2c1a34c06ec24e6297bef1bec0daa3fc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/40af8a0829224828ad0b2c21c9d48dcd] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=60.2 K 2023-04-18 18:55:56,094 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 26d5eac8fa074e59b6e787df250caa1e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=833, earliestPutTs=1681844153843 2023-04-18 18:55:56,094 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting c5254c10f7044f5c90d93e197a28de81, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=833, earliestPutTs=1681844153843 2023-04-18 18:55:56,094 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 372b2d4df0434946a45bf31f8fe38ff9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=851, earliestPutTs=1681844153898 2023-04-18 18:55:56,094 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 98f05764e9834fc091d29eb762698cba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=851, earliestPutTs=1681844153898 2023-04-18 18:55:56,094 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting db53449058b24fd38cfc70e1abe638b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=868, earliestPutTs=1681844154032 2023-04-18 18:55:56,095 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting e1134d06c3ff4fd39e1701b85d472645, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=868, earliestPutTs=1681844154032 2023-04-18 18:55:56,095 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting a11eb6bfe0904a67a8c6ffe851efc28f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=887, earliestPutTs=1681844155195 2023-04-18 18:55:56,096 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 2c1a34c06ec24e6297bef1bec0daa3fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=887, earliestPutTs=1681844155195 2023-04-18 18:55:56,096 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 433f2d2cf69e477d868105bd1903fae1, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=902, earliestPutTs=1681844155519 2023-04-18 18:55:56,096 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:56,096 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:56,096 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:56,096 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:56,096 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 40af8a0829224828ad0b2c21c9d48dcd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=902, earliestPutTs=1681844155527 2023-04-18 18:55:56,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:56,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:56,096 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:56,096 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:56,096 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,096 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:56,097 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:56,097 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:56,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:56,097 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:56,097 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:56,104 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#A#compaction#464 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:56,105 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#B#compaction#465 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:56,197 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:56,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:56,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:56,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:56,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:56,200 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:56,200 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:56,200 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:56,201 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:56,201 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:56,201 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:56,202 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,202 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:56,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:56,202 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,202 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:56,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:56,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:56,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:56,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:56,203 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:56,204 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:56,204 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:56,204 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-18 18:55:56,204 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:56,204 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:56,205 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:56,205 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:56,205 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:56,205 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:56,213 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=919 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/14103793e82c4c89a10f54a6169eb09a 2023-04-18 18:55:56,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:56,304 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:56,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844216317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844216317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844216317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844216327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844216327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844216433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844216433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844216433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844216433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844216436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,520 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/7c51936041d7463d857a2b1a16c0c374 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/7c51936041d7463d857a2b1a16c0c374 2023-04-18 18:55:56,522 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/7f722600491849fe871964d6b51e905a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/7f722600491849fe871964d6b51e905a 2023-04-18 18:55:56,525 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/B of 814b3c470b4969ca1c95de968f5dec66 into 7c51936041d7463d857a2b1a16c0c374(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:56,525 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:56,525 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/B, priority=11, startTime=1681844156091; duration=0sec 2023-04-18 18:55:56,525 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:56,525 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:55:56,527 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61646 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:55:56,527 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/C is initiating minor compaction (all files) 2023-04-18 18:55:56,527 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/C in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:56,527 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/fdf754176e1842be887864ec041b8566, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f7b60a88bda5418d8e9570881a0835d5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/bb0354d8e09c48889501d4442d7be77f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/6bae45fc63cf405594aafaf79499ffa3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/579f859a4748401db218b7567108b53a] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=60.2 K 2023-04-18 18:55:56,527 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting fdf754176e1842be887864ec041b8566, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=833, earliestPutTs=1681844153843 2023-04-18 18:55:56,528 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting f7b60a88bda5418d8e9570881a0835d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=851, earliestPutTs=1681844153898 2023-04-18 18:55:56,528 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting bb0354d8e09c48889501d4442d7be77f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=868, earliestPutTs=1681844154032 2023-04-18 18:55:56,528 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/A of 814b3c470b4969ca1c95de968f5dec66 into 7f722600491849fe871964d6b51e905a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:56,529 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:56,529 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/A, priority=11, startTime=1681844156091; duration=0sec 2023-04-18 18:55:56,529 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 6bae45fc63cf405594aafaf79499ffa3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=887, earliestPutTs=1681844155195 2023-04-18 18:55:56,529 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:56,529 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 579f859a4748401db218b7567108b53a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=902, earliestPutTs=1681844155527 2023-04-18 18:55:56,535 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#C#compaction#468 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:56,543 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/3bcf86f2e7294312bcc39671d0d0fdb5 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/3bcf86f2e7294312bcc39671d0d0fdb5 2023-04-18 18:55:56,547 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/C of 814b3c470b4969ca1c95de968f5dec66 into 3bcf86f2e7294312bcc39671d0d0fdb5(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:56,547 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:56,547 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/C, priority=11, startTime=1681844156091; duration=0sec 2023-04-18 18:55:56,548 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:56,623 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=919 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/47288e7124c34fcdbbf1ff9db0ce9814 2023-04-18 18:55:56,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844216637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844216637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844216639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844216639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844216643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844216942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844216943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844216945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844216946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:56,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:56,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844216950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,035 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=919 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/26d1fb3329b14ce688f6bbb3f1bdc8cd 2023-04-18 18:55:57,039 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/14103793e82c4c89a10f54a6169eb09a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/14103793e82c4c89a10f54a6169eb09a 2023-04-18 18:55:57,045 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/14103793e82c4c89a10f54a6169eb09a, entries=150, sequenceid=919, filesize=11.9 K 2023-04-18 18:55:57,045 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/47288e7124c34fcdbbf1ff9db0ce9814 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/47288e7124c34fcdbbf1ff9db0ce9814 2023-04-18 18:55:57,049 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/47288e7124c34fcdbbf1ff9db0ce9814, entries=150, sequenceid=919, filesize=11.9 K 2023-04-18 18:55:57,049 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/26d1fb3329b14ce688f6bbb3f1bdc8cd as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/26d1fb3329b14ce688f6bbb3f1bdc8cd 2023-04-18 18:55:57,053 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/26d1fb3329b14ce688f6bbb3f1bdc8cd, entries=150, sequenceid=919, filesize=11.9 K 2023-04-18 18:55:57,054 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 814b3c470b4969ca1c95de968f5dec66 in 850ms, sequenceid=919, compaction requested=false 2023-04-18 18:55:57,054 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:57,054 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:57,054 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:57,054 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:57,054 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,054 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:57,054 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:57,056 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,056 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:57,056 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:57,056 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:57,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:57,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:57,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79529f8[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:57,059 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:57,059 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:57,061 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:57,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:57,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:57,061 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:57,061 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:57,061 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:57,062 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,062 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:57,068 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:57,068 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:57,068 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,068 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:57,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:57,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:57,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:57,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:57,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:57,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:57,071 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:57,071 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:57,071 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:57,071 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:57,071 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:57,073 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:57,073 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:57,073 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:57,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:57,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:57,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:57,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:57,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:57,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:57,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:57,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:57,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:57,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:57,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:57,075 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:57,076 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:57,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:57,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:57,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:57,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:57,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:57,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:57,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:57,078 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:57,078 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:57,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:57,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:57,078 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:57,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:57,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:57,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:57,178 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:57,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:57,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:57,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:57,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:57,182 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:57,182 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:57,182 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:57,182 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:57,183 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:57,183 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:57,184 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:57,184 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:57,184 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,184 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:57,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:57,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:57,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:57,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:57,185 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:57,185 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:57,185 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:57,185 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:57,185 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:57,187 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:57,187 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:57,187 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-18 18:55:57,187 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:57,187 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:57,187 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:57,187 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:57,188 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:57,188 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:57,201 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=937 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/da817eeb8f2b421ab98879d60e684650 2023-04-18 18:55:57,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:57,451 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:57,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:57,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844217464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:57,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844217467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:57,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844217468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:57,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844217468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:57,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844217470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:57,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844217572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:57,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844217572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:57,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844217575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:57,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844217575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,581 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:57,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844217577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,611 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=937 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/edcd2565a4b147b593f7d0857ac40e63 2023-04-18 18:55:57,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:57,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844217777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:57,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844217777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:57,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844217778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:57,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844217778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:57,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:57,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844217782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,023 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=937 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/176e087ef2094ec7adc0024f26039d4e 2023-04-18 18:55:58,027 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/da817eeb8f2b421ab98879d60e684650 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/da817eeb8f2b421ab98879d60e684650 2023-04-18 18:55:58,030 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/da817eeb8f2b421ab98879d60e684650, entries=150, sequenceid=937, filesize=11.9 K 2023-04-18 18:55:58,031 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/edcd2565a4b147b593f7d0857ac40e63 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/edcd2565a4b147b593f7d0857ac40e63 2023-04-18 18:55:58,034 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/edcd2565a4b147b593f7d0857ac40e63, entries=150, sequenceid=937, filesize=11.9 K 2023-04-18 18:55:58,035 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/176e087ef2094ec7adc0024f26039d4e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/176e087ef2094ec7adc0024f26039d4e 2023-04-18 18:55:58,038 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/176e087ef2094ec7adc0024f26039d4e, entries=150, sequenceid=937, filesize=11.9 K 2023-04-18 18:55:58,039 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 814b3c470b4969ca1c95de968f5dec66 in 852ms, sequenceid=937, compaction requested=true 2023-04-18 18:55:58,039 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:58,039 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:58,039 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:58,039 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:58,039 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,039 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:58,039 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:58,042 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,042 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:58,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:58,043 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,043 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:58,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:58,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:58,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fd29e71[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:58,045 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:58,045 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,047 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,047 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:58,047 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:58,047 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:58,047 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,047 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:58,048 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:58,048 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:58,048 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,048 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:58,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:58,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:58,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:58,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:58,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:58,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:58,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:58,051 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:58,051 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:58,051 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:58,051 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:58,051 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:58,052 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,052 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:58,052 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,053 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:58,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:58,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:58,054 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:58,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:58,057 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:58,057 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:58,057 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:58,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:58,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:58,057 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:58,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:58,057 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:58,057 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:58,057 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,057 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,057 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,057 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:58,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:58,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:58,057 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:58,058 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:58,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:58,091 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-18 18:55:58,092 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:58,092 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:58,092 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:58,092 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:58,092 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:58,092 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:58,099 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=956 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/2812200484774f3db8280ac60507200b 2023-04-18 18:55:58,109 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=956 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/ffacb2b6c90848838734a44a4106c9b8 2023-04-18 18:55:58,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844218104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844218106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844218106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844218107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844218108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,158 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:58,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:58,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:58,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:58,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:58,161 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:58,161 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:58,161 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:58,161 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:58,161 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,161 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:58,165 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:58,165 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:58,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:58,166 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,166 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:58,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:58,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:58,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:58,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:58,167 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:58,167 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:58,167 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:58,167 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:58,167 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:58,167 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:58,167 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:58,167 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:58,167 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:58,167 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:58,167 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:58,167 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,167 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:58,167 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:58,169 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,169 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:58,169 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,169 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:58,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:58,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6633f6e6[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:58,170 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:58,170 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,171 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,171 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:58,171 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:58,171 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,172 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:58,171 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:58,174 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,174 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:58,174 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:58,174 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:58,177 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:58,177 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:58,177 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:58,177 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:58,177 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:58,178 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,178 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,178 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:58,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:58,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:58,179 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,180 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:58,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:58,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,182 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:58,182 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:58,182 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:58,183 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:58,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:58,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:58,183 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:58,183 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:58,183 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,183 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,183 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,183 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,183 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:58,183 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:58,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:58,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844218214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844218215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844218215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844218216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844218216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,283 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:58,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:58,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:58,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:58,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:58,288 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:58,288 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:58,288 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:58,288 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:58,289 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,289 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:58,290 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:58,290 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:58,291 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,291 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:58,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:58,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:58,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:58,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:58,292 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:58,292 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:58,292 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:58,292 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:58,293 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:58,293 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:58,293 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:58,293 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:58,293 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:58,293 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:58,293 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:58,293 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,293 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:58,293 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:58,296 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,296 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:58,296 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,296 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:58,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:58,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34fe1685[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:58,298 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:58,298 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,299 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,299 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:58,299 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:58,299 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:58,301 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:58,301 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:58,301 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,301 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:58,301 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:58,301 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:58,301 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:58,301 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:58,301 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:58,301 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:58,302 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:58,303 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,303 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,303 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:58,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:58,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:58,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:58,304 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:58,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:58,305 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:58,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:58,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,311 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,311 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:58,311 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:58,311 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:58,311 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:58,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:58,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:58,311 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:58,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:58,311 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:58,312 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:58,312 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:58,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:58,311 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:58,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:58,312 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,312 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,412 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:58,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:58,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:58,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:58,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:58,415 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:58,415 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:58,415 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:58,415 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:58,416 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,416 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:58,417 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:58,417 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:58,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:58,418 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,418 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:58,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:58,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:58,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:58,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:58,419 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:58,419 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:58,419 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:58,419 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:58,421 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,421 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:58,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:58,421 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,421 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:58,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:58,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:58,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:58,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:58,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:58,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a0a330b[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:58,423 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:58,423 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,424 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,424 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:58,424 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:58,424 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:58,424 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,425 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:58,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844218420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844218421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844218421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844218421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844218422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,432 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,432 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:58,432 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:58,432 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:58,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:58,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:58,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:58,434 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:58,434 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:58,434 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:58,435 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:58,435 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:58,436 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,436 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:58,436 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:58,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:58,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:58,437 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:58,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:58,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:58,438 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:58,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:58,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,441 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,441 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:58,441 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:58,441 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:58,441 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:58,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:58,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:58,441 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,441 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,441 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:58,441 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:58,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:58,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:58,441 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:58,442 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:58,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:58,441 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:58,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:58,517 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=956 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/49628cafe1cb4e608dd4920a133eb8d9 2023-04-18 18:55:58,521 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/2812200484774f3db8280ac60507200b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/2812200484774f3db8280ac60507200b 2023-04-18 18:55:58,524 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/2812200484774f3db8280ac60507200b, entries=250, sequenceid=956, filesize=16.7 K 2023-04-18 18:55:58,525 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/ffacb2b6c90848838734a44a4106c9b8 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ffacb2b6c90848838734a44a4106c9b8 2023-04-18 18:55:58,528 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ffacb2b6c90848838734a44a4106c9b8, entries=150, sequenceid=956, filesize=11.9 K 2023-04-18 18:55:58,529 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/49628cafe1cb4e608dd4920a133eb8d9 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/49628cafe1cb4e608dd4920a133eb8d9 2023-04-18 18:55:58,532 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/49628cafe1cb4e608dd4920a133eb8d9, entries=150, sequenceid=956, filesize=11.9 K 2023-04-18 18:55:58,533 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 814b3c470b4969ca1c95de968f5dec66 in 442ms, sequenceid=956, compaction requested=true 2023-04-18 18:55:58,533 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:58,533 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:58,533 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:55:58,533 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:55:58,533 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:58,533 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:58,539 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54510 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:55:58,539 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/A is initiating minor compaction (all files) 2023-04-18 18:55:58,539 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/A in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:58,539 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/7f722600491849fe871964d6b51e905a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/14103793e82c4c89a10f54a6169eb09a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/da817eeb8f2b421ab98879d60e684650, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/2812200484774f3db8280ac60507200b] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=53.2 K 2023-04-18 18:55:58,539 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:55:58,539 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/B is initiating minor compaction (all files) 2023-04-18 18:55:58,539 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/B in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:58,539 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/7c51936041d7463d857a2b1a16c0c374, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/47288e7124c34fcdbbf1ff9db0ce9814, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/edcd2565a4b147b593f7d0857ac40e63, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ffacb2b6c90848838734a44a4106c9b8] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=48.5 K 2023-04-18 18:55:58,540 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 7f722600491849fe871964d6b51e905a, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=902, earliestPutTs=1681844155527 2023-04-18 18:55:58,540 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 7c51936041d7463d857a2b1a16c0c374, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=902, earliestPutTs=1681844155527 2023-04-18 18:55:58,540 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 14103793e82c4c89a10f54a6169eb09a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=919, earliestPutTs=1681844155661 2023-04-18 18:55:58,540 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 47288e7124c34fcdbbf1ff9db0ce9814, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=919, earliestPutTs=1681844155661 2023-04-18 18:55:58,540 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting da817eeb8f2b421ab98879d60e684650, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=937, earliestPutTs=1681844156316 2023-04-18 18:55:58,542 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:58,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:58,542 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting edcd2565a4b147b593f7d0857ac40e63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=937, earliestPutTs=1681844156316 2023-04-18 18:55:58,543 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 2812200484774f3db8280ac60507200b, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1681844157461 2023-04-18 18:55:58,543 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting ffacb2b6c90848838734a44a4106c9b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1681844157461 2023-04-18 18:55:58,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:58,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:58,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:58,545 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:58,545 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:58,545 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:58,545 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:58,546 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,546 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:58,547 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:58,547 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:58,548 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,548 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:58,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:58,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:58,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:58,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:58,550 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:58,550 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:58,550 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:58,551 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:58,551 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:58,551 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 18:55:58,551 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:58,551 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:58,551 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:58,551 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:58,551 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:58,551 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:58,558 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#A#compaction#476 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:58,561 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#B#compaction#477 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:58,584 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=969 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/2942ac20b7a54451b09e7feb8c40b212 2023-04-18 18:55:58,593 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/debacd8c5cc54426bb8e2d540284b216 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/debacd8c5cc54426bb8e2d540284b216 2023-04-18 18:55:58,599 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/B of 814b3c470b4969ca1c95de968f5dec66 into debacd8c5cc54426bb8e2d540284b216(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:58,599 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:58,599 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/B, priority=12, startTime=1681844158533; duration=0sec 2023-04-18 18:55:58,599 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:58,599 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:55:58,602 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:55:58,602 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/C is initiating minor compaction (all files) 2023-04-18 18:55:58,602 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/C in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:58,602 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/3bcf86f2e7294312bcc39671d0d0fdb5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/26d1fb3329b14ce688f6bbb3f1bdc8cd, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/176e087ef2094ec7adc0024f26039d4e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/49628cafe1cb4e608dd4920a133eb8d9] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=48.5 K 2023-04-18 18:55:58,603 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 3bcf86f2e7294312bcc39671d0d0fdb5, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=902, earliestPutTs=1681844155527 2023-04-18 18:55:58,603 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=969 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/500dfdd33f4e42aaa803bf07b8a36446 2023-04-18 18:55:58,604 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 26d1fb3329b14ce688f6bbb3f1bdc8cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=919, earliestPutTs=1681844155661 2023-04-18 18:55:58,604 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 176e087ef2094ec7adc0024f26039d4e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=937, earliestPutTs=1681844156316 2023-04-18 18:55:58,604 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 49628cafe1cb4e608dd4920a133eb8d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1681844157461 2023-04-18 18:55:58,618 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#C#compaction#481 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:58,630 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/24bdfcbfaecd45868696a8d2c1690f0c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/24bdfcbfaecd45868696a8d2c1690f0c 2023-04-18 18:55:58,635 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/C of 814b3c470b4969ca1c95de968f5dec66 into 24bdfcbfaecd45868696a8d2c1690f0c(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:58,635 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:58,635 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/C, priority=12, startTime=1681844158533; duration=0sec 2023-04-18 18:55:58,635 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:58,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:58,734 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:58,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844218752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844218753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844218753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844218754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844218754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844218855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844218860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844218860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844218860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:58,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844218861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:58,978 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/0161a97118f74cfa8d01062c97795041 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/0161a97118f74cfa8d01062c97795041 2023-04-18 18:55:58,986 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/A of 814b3c470b4969ca1c95de968f5dec66 into 0161a97118f74cfa8d01062c97795041(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:58,986 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:58,986 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/A, priority=12, startTime=1681844158533; duration=0sec 2023-04-18 18:55:58,986 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:59,021 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=969 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/ade1ebe765dc432685d5e84c3bad704d 2023-04-18 18:55:59,027 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/2942ac20b7a54451b09e7feb8c40b212 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/2942ac20b7a54451b09e7feb8c40b212 2023-04-18 18:55:59,032 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/2942ac20b7a54451b09e7feb8c40b212, entries=150, sequenceid=969, filesize=11.9 K 2023-04-18 18:55:59,032 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/500dfdd33f4e42aaa803bf07b8a36446 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/500dfdd33f4e42aaa803bf07b8a36446 2023-04-18 18:55:59,036 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/500dfdd33f4e42aaa803bf07b8a36446, entries=150, sequenceid=969, filesize=11.9 K 2023-04-18 18:55:59,036 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/ade1ebe765dc432685d5e84c3bad704d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/ade1ebe765dc432685d5e84c3bad704d 2023-04-18 18:55:59,040 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/ade1ebe765dc432685d5e84c3bad704d, entries=150, sequenceid=969, filesize=11.9 K 2023-04-18 18:55:59,040 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 814b3c470b4969ca1c95de968f5dec66 in 489ms, sequenceid=969, compaction requested=false 2023-04-18 18:55:59,040 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:59,040 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:59,040 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:59,040 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:59,041 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,041 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:59,041 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:59,046 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,046 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:59,047 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,047 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:59,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:59,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6150172d[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:59,048 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:59,048 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,049 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,050 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:59,050 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:59,050 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,050 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:59,050 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:59,051 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:59,051 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:59,051 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,051 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:59,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:59,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:59,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:59,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:59,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:59,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:59,053 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:59,054 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:59,054 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:59,054 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:59,054 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:59,056 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,056 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:59,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,056 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:59,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:59,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:59,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:59,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:59,057 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,060 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,063 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,063 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,063 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,063 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:59,063 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:59,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:59,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:59,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:59,064 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,064 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,064 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 18:55:59,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:59,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:59,064 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,064 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,064 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:59,064 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:59,064 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:59,064 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:59,064 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:59,064 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:59,064 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:59,065 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:59,064 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:59,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:59,072 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=992 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/07b91115b91a4b3f828538bdea808591 2023-04-18 18:55:59,080 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=992 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/37aed5b73362471ab4912aa650df52c1 2023-04-18 18:55:59,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:59,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844219075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:59,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844219075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:59,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844219076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:59,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844219080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:59,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844219081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,091 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=992 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/7f1828bcafa141959f50fa11a790d954 2023-04-18 18:55:59,094 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/07b91115b91a4b3f828538bdea808591 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/07b91115b91a4b3f828538bdea808591 2023-04-18 18:55:59,098 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/07b91115b91a4b3f828538bdea808591, entries=200, sequenceid=992, filesize=14.3 K 2023-04-18 18:55:59,098 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/37aed5b73362471ab4912aa650df52c1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/37aed5b73362471ab4912aa650df52c1 2023-04-18 18:55:59,102 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/37aed5b73362471ab4912aa650df52c1, entries=150, sequenceid=992, filesize=11.9 K 2023-04-18 18:55:59,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/7f1828bcafa141959f50fa11a790d954 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/7f1828bcafa141959f50fa11a790d954 2023-04-18 18:55:59,106 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/7f1828bcafa141959f50fa11a790d954, entries=150, sequenceid=992, filesize=11.9 K 2023-04-18 18:55:59,107 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 814b3c470b4969ca1c95de968f5dec66 in 43ms, sequenceid=992, compaction requested=true 2023-04-18 18:55:59,107 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:59,107 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:59,107 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:59,107 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:59,107 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:59,107 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:59,111 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40020 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:59,111 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:59,111 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/A is initiating minor compaction (all files) 2023-04-18 18:55:59,111 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/B is initiating minor compaction (all files) 2023-04-18 18:55:59,111 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/A in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:59,111 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/B in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:59,111 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/0161a97118f74cfa8d01062c97795041, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/2942ac20b7a54451b09e7feb8c40b212, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/07b91115b91a4b3f828538bdea808591] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=39.1 K 2023-04-18 18:55:59,111 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/debacd8c5cc54426bb8e2d540284b216, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/500dfdd33f4e42aaa803bf07b8a36446, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/37aed5b73362471ab4912aa650df52c1] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=36.7 K 2023-04-18 18:55:59,112 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 0161a97118f74cfa8d01062c97795041, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1681844157461 2023-04-18 18:55:59,112 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting debacd8c5cc54426bb8e2d540284b216, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1681844157461 2023-04-18 18:55:59,112 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 2942ac20b7a54451b09e7feb8c40b212, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=969, earliestPutTs=1681844158104 2023-04-18 18:55:59,112 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 500dfdd33f4e42aaa803bf07b8a36446, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=969, earliestPutTs=1681844158104 2023-04-18 18:55:59,112 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 07b91115b91a4b3f828538bdea808591, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=992, earliestPutTs=1681844158735 2023-04-18 18:55:59,113 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 37aed5b73362471ab4912aa650df52c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=992, earliestPutTs=1681844158752 2023-04-18 18:55:59,123 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#A#compaction#485 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:59,127 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#B#compaction#486 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:59,144 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/f97af59c9f524916a35a9985e257ed92 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/f97af59c9f524916a35a9985e257ed92 2023-04-18 18:55:59,145 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/980493ba9365453faf67d480c7cd062f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/980493ba9365453faf67d480c7cd062f 2023-04-18 18:55:59,158 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/B of 814b3c470b4969ca1c95de968f5dec66 into f97af59c9f524916a35a9985e257ed92(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:59,158 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:59,158 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/B, priority=13, startTime=1681844159107; duration=0sec 2023-04-18 18:55:59,158 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:55:59,158 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:55:59,159 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/A of 814b3c470b4969ca1c95de968f5dec66 into 980493ba9365453faf67d480c7cd062f(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:59,159 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:59,159 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/A, priority=13, startTime=1681844159107; duration=0sec 2023-04-18 18:55:59,159 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:59,164 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:55:59,164 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/C is initiating minor compaction (all files) 2023-04-18 18:55:59,164 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/C in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:59,164 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/24bdfcbfaecd45868696a8d2c1690f0c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/ade1ebe765dc432685d5e84c3bad704d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/7f1828bcafa141959f50fa11a790d954] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=36.7 K 2023-04-18 18:55:59,165 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:59,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:59,166 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 24bdfcbfaecd45868696a8d2c1690f0c, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1681844157461 2023-04-18 18:55:59,166 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting ade1ebe765dc432685d5e84c3bad704d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=969, earliestPutTs=1681844158104 2023-04-18 18:55:59,167 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 7f1828bcafa141959f50fa11a790d954, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=992, earliestPutTs=1681844158752 2023-04-18 18:55:59,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:59,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:59,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:59,168 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:59,168 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:59,168 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:59,168 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:59,169 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,169 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,170 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:59,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:59,170 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:59,171 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,171 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:59,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,172 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:59,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:59,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:59,173 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:59,173 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:59,173 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:59,173 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:59,173 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:59,173 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:59,173 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:59,173 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-18 18:55:59,173 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:59,174 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:59,174 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:59,174 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:59,174 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:59,174 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:55:59,183 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#C#compaction#487 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:55:59,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:59,190 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:59,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:59,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844219208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,215 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1006 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/9da3a4951aca4b78b0af81083c45c2b9 2023-04-18 18:55:59,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:59,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844219210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,217 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/e2dc6cf456834b07a81aee271c88e8f1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/e2dc6cf456834b07a81aee271c88e8f1 2023-04-18 18:55:59,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:59,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844219211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:59,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844219211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:59,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844219212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,221 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/C of 814b3c470b4969ca1c95de968f5dec66 into e2dc6cf456834b07a81aee271c88e8f1(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:55:59,221 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:59,221 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/C, priority=13, startTime=1681844159107; duration=0sec 2023-04-18 18:55:59,222 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:55:59,225 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1006 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/a3cf1b44f9bb4738bdd4ed715be29c70 2023-04-18 18:55:59,236 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1006 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/af648e86e37641009c4770bfdff8fc1d 2023-04-18 18:55:59,241 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/9da3a4951aca4b78b0af81083c45c2b9 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/9da3a4951aca4b78b0af81083c45c2b9 2023-04-18 18:55:59,245 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/9da3a4951aca4b78b0af81083c45c2b9, entries=150, sequenceid=1006, filesize=11.9 K 2023-04-18 18:55:59,245 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/a3cf1b44f9bb4738bdd4ed715be29c70 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/a3cf1b44f9bb4738bdd4ed715be29c70 2023-04-18 18:55:59,249 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/a3cf1b44f9bb4738bdd4ed715be29c70, entries=150, sequenceid=1006, filesize=11.9 K 2023-04-18 18:55:59,250 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/af648e86e37641009c4770bfdff8fc1d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/af648e86e37641009c4770bfdff8fc1d 2023-04-18 18:55:59,253 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/af648e86e37641009c4770bfdff8fc1d, entries=150, sequenceid=1006, filesize=11.9 K 2023-04-18 18:55:59,254 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 814b3c470b4969ca1c95de968f5dec66 in 81ms, sequenceid=1006, compaction requested=false 2023-04-18 18:55:59,254 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:55:59,254 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:59,254 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:59,254 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:59,254 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,254 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:59,254 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:59,256 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,256 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:59,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:59,256 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,256 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:59,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:59,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:59,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@182d4927[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:59,258 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:59,258 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,263 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,263 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,263 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:59,263 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:59,263 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:59,264 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:59,264 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,264 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:59,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:59,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:59,264 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:59,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:59,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:59,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:59,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:59,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:59,270 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:59,270 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:59,270 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:59,270 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:59,270 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:59,272 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,272 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:59,272 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:59,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:59,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:59,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:59,273 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,274 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,276 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:59,276 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:59,276 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,277 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,276 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:59,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:59,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:59,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:59,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:59,277 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,277 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,277 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,277 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,277 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:59,278 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:59,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:59,277 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:59,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:59,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:55:59,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844159202, max timestamp=1681844159314Num uniques 859; 2023-04-18 18:55:59,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844159202, max timestamp=1681844159314Num uniques 859; 2023-04-18 18:55:59,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:55:59,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844159202, max timestamp=1681844159314Num uniques 859; 2023-04-18 18:55:59,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:55:59,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:55:59,317 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=58.14 KB heapSize=164.17 KB 2023-04-18 18:55:59,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:55:59,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:59,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:55:59,318 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:59,318 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:55:59,318 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:55:59,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681844159325, max timestamp=1681844159347Num uniques 859; 2023-04-18 18:55:59,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 18:55:59,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681844159325, max timestamp=1681844159347Num uniques 859; 2023-04-18 18:55:59,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 18:55:59,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681844159325, max timestamp=1681844159347Num uniques 859; 2023-04-18 18:55:59,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 18:55:59,378 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:59,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:59,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:59,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1681844159369, max timestamp=1681844159386Num uniques 1118; 2023-04-18 18:55:59,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1681844159369, max timestamp=1681844159386Num uniques 1118; 2023-04-18 18:55:59,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=16848, suffix heap size=144240,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=368 2023-04-18 18:55:59,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1681844159369, max timestamp=1681844159386Num uniques 1118; 2023-04-18 18:55:59,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=16848, suffix heap size=144240,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=368 2023-04-18 18:55:59,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=16848, suffix heap size=144240,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=368 2023-04-18 18:55:59,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:59,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:59,392 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:59,392 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:59,392 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:59,392 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:59,393 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,393 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,395 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:59,395 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,396 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,396 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:59,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:59,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:59,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:59,400 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:59,400 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:59,400 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:59,409 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:59,410 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:59,410 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:59,410 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:59,410 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:59,410 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:59,410 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,410 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:59,410 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:59,412 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,412 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:59,412 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,412 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:59,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:59,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:59,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:59,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c233e99[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:59,414 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:59,414 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,416 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,416 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,416 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:59,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,416 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:59,416 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:59,416 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:59,418 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:59,418 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:59,418 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,418 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:59,420 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:59,421 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:59,421 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:59,421 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:59,421 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:59,424 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,425 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:59,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,424 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:59,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:59,425 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,427 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681844159422, max timestamp=1681844159429Num uniques 1227; 2023-04-18 18:55:59,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=10392, suffix heap size=155576,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=227 2023-04-18 18:55:59,432 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:59,432 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:59,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:59,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:59,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:59,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:59,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681844159422, max timestamp=1681844159429Num uniques 1227; 2023-04-18 18:55:59,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=10392, suffix heap size=155576,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=227 2023-04-18 18:55:59,433 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,433 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,433 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,433 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,433 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,433 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,433 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,433 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:59,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:59,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:59,433 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:59,433 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:59,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681844159422, max timestamp=1681844159429Num uniques 1227; 2023-04-18 18:55:59,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=10392, suffix heap size=155576,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=227 2023-04-18 18:55:59,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:59,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681844159455, max timestamp=1681844159469Num uniques 1086; 2023-04-18 18:55:59,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-18 18:55:59,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681844159455, max timestamp=1681844159469Num uniques 1086; 2023-04-18 18:55:59,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-18 18:55:59,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681844159455, max timestamp=1681844159469Num uniques 1086; 2023-04-18 18:55:59,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-18 18:55:59,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:59,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844219501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681844159495, max timestamp=1681844159506Num uniques 1145; 2023-04-18 18:55:59,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-18 18:55:59,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681844159495, max timestamp=1681844159506Num uniques 1145; 2023-04-18 18:55:59,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-18 18:55:59,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681844159495, max timestamp=1681844159506Num uniques 1145; 2023-04-18 18:55:59,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-18 18:55:59,533 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:59,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:59,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:59,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:59,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:59,536 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:59,536 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:59,536 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:59,536 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:59,537 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,537 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,538 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,538 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:59,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:59,538 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,538 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:59,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:59,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:59,539 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:59,540 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:59,540 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:59,540 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:59,540 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:59,540 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:59,540 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:59,540 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:59,540 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:59,540 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:59,540 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:59,541 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,541 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:59,541 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,542 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:59,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:59,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@177acc7d[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:59,544 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:59,544 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,547 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,547 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:59,547 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:59,547 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:59,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:55:59,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844219543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681844159537, max timestamp=1681844159546Num uniques 1154; 2023-04-18 18:55:59,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681844159537, max timestamp=1681844159546Num uniques 1154; 2023-04-18 18:55:59,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681844159537, max timestamp=1681844159546Num uniques 1154; 2023-04-18 18:55:59,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-04-18 18:55:59,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-04-18 18:55:59,549 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,550 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:59,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-04-18 18:55:59,554 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:59,554 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:59,554 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:59,554 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:59,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:59,557 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:59,558 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:59,558 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:59,558 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:59,558 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:59,559 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,559 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:59,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:59,559 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:59,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:59,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:59,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:59,560 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:59,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,561 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:59,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,564 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,564 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,564 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:59,564 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:59,564 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:59,564 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:55:59,564 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:59,564 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:59,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:59,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:59,564 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,564 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,565 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:59,565 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:59,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:59,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:59,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681844159579, max timestamp=1681844159596Num uniques 1013; 2023-04-18 18:55:59,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681844159579, max timestamp=1681844159596Num uniques 1013; 2023-04-18 18:55:59,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-18 18:55:59,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-18 18:55:59,599 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681844159579, max timestamp=1681844159596Num uniques 1013; 2023-04-18 18:55:59,599 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-18 18:55:59,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681844159613, max timestamp=1681844159640Num uniques 1072; 2023-04-18 18:55:59,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-04-18 18:55:59,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681844159613, max timestamp=1681844159640Num uniques 1072; 2023-04-18 18:55:59,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-04-18 18:55:59,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681844159613, max timestamp=1681844159640Num uniques 1072; 2023-04-18 18:55:59,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-04-18 18:55:59,665 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:59,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:59,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:59,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:59,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:59,668 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:59,668 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:59,668 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:59,668 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:59,668 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,669 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,675 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,675 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:59,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:59,675 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,675 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:59,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:59,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:59,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:59,676 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:59,676 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:59,676 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:59,677 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:59,677 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:59,677 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:59,677 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:59,677 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:59,677 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:59,677 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:59,677 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:59,677 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,677 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:59,677 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:59,679 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,679 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:59,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:59,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:59,679 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,679 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:59,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1681844159668, max timestamp=1681844159679Num uniques 1131; 2023-04-18 18:55:59,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=12866, suffix heap size=145592,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=281 2023-04-18 18:55:59,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1681844159668, max timestamp=1681844159679Num uniques 1131; 2023-04-18 18:55:59,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1681844159668, max timestamp=1681844159679Num uniques 1131; 2023-04-18 18:55:59,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=12866, suffix heap size=145592,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=281 2023-04-18 18:55:59,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=12866, suffix heap size=145592,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=281 2023-04-18 18:55:59,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:59,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:59,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:59,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76f248c0[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:59,685 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:59,685 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,686 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,686 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:59,686 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:59,686 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:59,686 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,686 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:59,687 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,688 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:59,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:59,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:59,688 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:59,688 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:59,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:59,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:59,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:59,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:59,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:59,691 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:59,691 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:59,691 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:59,691 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:59,691 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:59,694 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:59,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:59,694 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,694 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:59,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:59,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:59,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:59,695 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:59,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,696 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:59,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,699 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,699 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,699 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,699 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:59,699 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,699 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,699 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,699 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,699 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:59,699 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:59,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:59,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:59,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:59,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:59,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:59,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:59,700 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:59,700 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:59,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1681844159698, max timestamp=1681844159715Num uniques 1140; 2023-04-18 18:55:59,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=15570, suffix heap size=146528,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=340 2023-04-18 18:55:59,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1681844159698, max timestamp=1681844159715Num uniques 1140; 2023-04-18 18:55:59,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=15570, suffix heap size=146528,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=340 2023-04-18 18:55:59,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1681844159698, max timestamp=1681844159715Num uniques 1140; 2023-04-18 18:55:59,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=15570, suffix heap size=146528,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=340 2023-04-18 18:55:59,738 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1028 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/49e4709b8d3743a7a26dd4afd5fc3fdb 2023-04-18 18:55:59,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1681844159732, max timestamp=1681844159752Num uniques 1199; 2023-04-18 18:55:59,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=13694, suffix heap size=152664,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=299 2023-04-18 18:55:59,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1681844159732, max timestamp=1681844159752Num uniques 1199; 2023-04-18 18:55:59,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1681844159732, max timestamp=1681844159752Num uniques 1199; 2023-04-18 18:55:59,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=13694, suffix heap size=152664,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=299 2023-04-18 18:55:59,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=13694, suffix heap size=152664,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=299 2023-04-18 18:55:59,758 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1028 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/8e80d1fa52e64dc5b6a6fd0e502bda39 2023-04-18 18:55:59,800 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:59,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:59,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:59,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:59,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:59,804 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:59,804 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:59,804 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:59,804 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:59,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681844159788, max timestamp=1681844159799Num uniques 1158; 2023-04-18 18:55:59,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-04-18 18:55:59,805 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,805 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681844159788, max timestamp=1681844159799Num uniques 1158; 2023-04-18 18:55:59,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-04-18 18:55:59,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681844159788, max timestamp=1681844159799Num uniques 1158; 2023-04-18 18:55:59,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-04-18 18:55:59,808 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:59,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:59,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:59,808 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,808 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,808 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:59,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:59,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:59,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:59,809 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:59,809 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:59,809 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:59,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:59,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:59,812 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:59,812 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:59,812 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:59,812 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:59,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:59,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:59,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:59,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:59,814 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,814 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:59,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:59,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:59,815 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,815 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:59,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:59,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:59,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:59,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e44348[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:59,818 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:59,818 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,825 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,825 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:59,825 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:59,825 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:59,826 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,826 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:59,827 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:59,827 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:59,827 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:59,827 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:59,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:59,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:59,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:59,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:59,832 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:59,832 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:59,832 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:59,832 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:59,832 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:59,834 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,834 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,834 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:59,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:59,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:59,836 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:59,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,839 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:59,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,841 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:59,841 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:59,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:59,842 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:59,842 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,842 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,842 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,842 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,842 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,842 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,841 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:59,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:59,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:59,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:59,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:59,842 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:59,842 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:59,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681844159821, max timestamp=1681844159845Num uniques 1017; 2023-04-18 18:55:59,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681844159821, max timestamp=1681844159845Num uniques 1017; 2023-04-18 18:55:59,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=12222, suffix heap size=133736,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=267 2023-04-18 18:55:59,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=12222, suffix heap size=133736,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=267 2023-04-18 18:55:59,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681844159821, max timestamp=1681844159845Num uniques 1017; 2023-04-18 18:55:59,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-18 18:55:59,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1681844159875, max timestamp=1681844159890Num uniques 1126; 2023-04-18 18:55:59,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=10346, suffix heap size=145072,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=226 2023-04-18 18:55:59,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1681844159875, max timestamp=1681844159890Num uniques 1126; 2023-04-18 18:55:59,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=10346, suffix heap size=145072,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=226 2023-04-18 18:55:59,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1681844159875, max timestamp=1681844159890Num uniques 1026; 2023-04-18 18:55:59,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=10346, suffix heap size=134672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=226 2023-04-18 18:55:59,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1681844159914, max timestamp=1681844159926Num uniques 1085; 2023-04-18 18:55:59,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1681844159914, max timestamp=1681844159926Num uniques 1085; 2023-04-18 18:55:59,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=13050, suffix heap size=140808,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=285 2023-04-18 18:55:59,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1681844159914, max timestamp=1681844159926Num uniques 1085; 2023-04-18 18:55:59,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=13050, suffix heap size=140808,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=285 2023-04-18 18:55:59,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=13050, suffix heap size=140808,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=285 2023-04-18 18:55:59,942 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:55:59,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:55:59,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:55:59,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:55:59,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:55:59,946 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:55:59,947 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:59,947 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:55:59,947 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:55:59,947 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,947 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,954 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:59,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:59,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:59,954 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,954 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,954 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:55:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:55:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:55:59,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:55:59,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:55:59,957 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:55:59,957 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:55:59,957 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:55:59,957 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:55:59,957 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:55:59,957 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:59,957 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:55:59,957 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:55:59,957 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:55:59,957 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:55:59,957 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:55:59,957 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,957 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:55:59,957 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:55:59,959 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,959 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:59,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:59,959 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,959 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:55:59,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:59,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:59,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:59,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:55:59,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@417672bb[Count = 0] remaining members to acquire global barrier 2023-04-18 18:55:59,962 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:55:59,962 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,963 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,963 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:55:59,963 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:55:59,963 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:55:59,964 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,964 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:55:59,965 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:59,966 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:55:59,966 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:59,966 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:55:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:55:59,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:55:59,968 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:55:59,968 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:55:59,968 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:55:59,968 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:55:59,968 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:55:59,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681844159947, max timestamp=1681844159968Num uniques 1144; 2023-04-18 18:55:59,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=8884, suffix heap size=146944,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=194 2023-04-18 18:55:59,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681844159947, max timestamp=1681844159968Num uniques 1144; 2023-04-18 18:55:59,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=8884, suffix heap size=146944,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=194 2023-04-18 18:55:59,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681844159947, max timestamp=1681844159968Num uniques 1144; 2023-04-18 18:55:59,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=8884, suffix heap size=146944,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=194 2023-04-18 18:55:59,977 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:55:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:55:59,977 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,977 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:59,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:55:59,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:55:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:55:59,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,979 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:55:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:55:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,980 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,982 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,983 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,983 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:55:59,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:55:59,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:55:59,983 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:55:59,983 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:55:59,983 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:55:59,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:55:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:55:59,983 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:55:59,983 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:55:59,983 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,983 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:55:59,983 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:55:59,983 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:55:59,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:55:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:00,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681844159994, max timestamp=1681844160013Num uniques 1053; 2023-04-18 18:56:00,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-18 18:56:00,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681844159994, max timestamp=1681844160013Num uniques 1053; 2023-04-18 18:56:00,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681844159994, max timestamp=1681844160013Num uniques 1053; 2023-04-18 18:56:00,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-18 18:56:00,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-18 18:56:00,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1681844160037, max timestamp=1681844160050Num uniques 1012; 2023-04-18 18:56:00,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1681844160037, max timestamp=1681844160050Num uniques 1012; 2023-04-18 18:56:00,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-04-18 18:56:00,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1681844160037, max timestamp=1681844160050Num uniques 1012; 2023-04-18 18:56:00,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-04-18 18:56:00,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-04-18 18:56:00,083 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:00,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:00,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:00,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:00,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:00,088 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:00,088 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:00,088 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:00,089 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:00,089 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,089 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681844160077, max timestamp=1681844160089Num uniques 1071; 2023-04-18 18:56:00,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-04-18 18:56:00,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681844160077, max timestamp=1681844160089Num uniques 1071; 2023-04-18 18:56:00,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-04-18 18:56:00,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681844160077, max timestamp=1681844160089Num uniques 1071; 2023-04-18 18:56:00,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-04-18 18:56:00,094 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,094 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:00,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:00,094 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,094 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:00,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:00,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:00,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:00,096 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:00,096 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:00,096 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:00,096 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:00,096 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:00,096 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:00,096 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:00,096 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:00,096 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:00,096 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:00,096 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:00,097 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,097 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:00,097 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:00,099 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,099 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:00,100 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:00,100 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:00,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:00,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e57cc[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:00,101 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:00,102 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,104 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,104 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:00,104 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:00,104 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:00,104 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,104 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:00,105 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,105 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:00,105 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:00,106 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:00,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:00,108 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:00,108 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:00,108 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:00,108 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:00,108 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:00,109 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,110 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,110 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:00,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:00,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:00,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:00,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:00,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:00,110 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:00,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,114 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,118 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,118 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,118 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,118 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:00,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:00,118 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,118 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,118 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,118 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:00,118 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:00,118 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:00,118 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:00,118 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:00,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:00,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:00,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1681844160115, max timestamp=1681844160133Num uniques 1080; 2023-04-18 18:56:00,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=12820, suffix heap size=140288,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=280 2023-04-18 18:56:00,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1681844160115, max timestamp=1681844160133Num uniques 1080; 2023-04-18 18:56:00,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=12820, suffix heap size=140288,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=280 2023-04-18 18:56:00,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1681844160115, max timestamp=1681844160133Num uniques 1080; 2023-04-18 18:56:00,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=12820, suffix heap size=140288,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=280 2023-04-18 18:56:00,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1028 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/58b159a091924a68b4173c292fe9ed14 2023-04-18 18:56:00,173 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/49e4709b8d3743a7a26dd4afd5fc3fdb as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/49e4709b8d3743a7a26dd4afd5fc3fdb 2023-04-18 18:56:00,177 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/49e4709b8d3743a7a26dd4afd5fc3fdb, entries=200, sequenceid=1028, filesize=14.3 K 2023-04-18 18:56:00,178 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/8e80d1fa52e64dc5b6a6fd0e502bda39 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8e80d1fa52e64dc5b6a6fd0e502bda39 2023-04-18 18:56:00,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1681844160159, max timestamp=1681844160174Num uniques 1139; 2023-04-18 18:56:00,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=8654, suffix heap size=146424,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=189 2023-04-18 18:56:00,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1681844160159, max timestamp=1681844160174Num uniques 1139; 2023-04-18 18:56:00,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=8654, suffix heap size=146424,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=189 2023-04-18 18:56:00,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8e80d1fa52e64dc5b6a6fd0e502bda39, entries=150, sequenceid=1028, filesize=11.9 K 2023-04-18 18:56:00,181 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/58b159a091924a68b4173c292fe9ed14 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/58b159a091924a68b4173c292fe9ed14 2023-04-18 18:56:00,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1681844160159, max timestamp=1681844160174Num uniques 1139; 2023-04-18 18:56:00,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=8654, suffix heap size=146424,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=189 2023-04-18 18:56:00,185 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/58b159a091924a68b4173c292fe9ed14, entries=150, sequenceid=1028, filesize=11.9 K 2023-04-18 18:56:00,187 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=33.54 KB/34350 for 814b3c470b4969ca1c95de968f5dec66 in 870ms, sequenceid=1028, compaction requested=true 2023-04-18 18:56:00,187 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:00,187 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:00,187 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-18 18:56:00,187 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-18 18:56:00,187 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:00,187 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:00,188 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40122 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:00,189 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/A is initiating minor compaction (all files) 2023-04-18 18:56:00,189 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/A in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:00,189 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/980493ba9365453faf67d480c7cd062f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/9da3a4951aca4b78b0af81083c45c2b9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/49e4709b8d3743a7a26dd4afd5fc3fdb] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=39.2 K 2023-04-18 18:56:00,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:56:00,189 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=53.67 KB heapSize=143.11 KB 2023-04-18 18:56:00,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:56:00,190 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:00,190 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:56:00,190 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:00,190 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:56:00,190 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:00,191 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:00,191 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/B is initiating minor compaction (all files) 2023-04-18 18:56:00,191 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/B in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:00,191 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/f97af59c9f524916a35a9985e257ed92, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/a3cf1b44f9bb4738bdd4ed715be29c70, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8e80d1fa52e64dc5b6a6fd0e502bda39] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=36.8 K 2023-04-18 18:56:00,193 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 980493ba9365453faf67d480c7cd062f, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=992, earliestPutTs=1681844158752 2023-04-18 18:56:00,194 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting f97af59c9f524916a35a9985e257ed92, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=992, earliestPutTs=1681844158752 2023-04-18 18:56:00,194 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 9da3a4951aca4b78b0af81083c45c2b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1006, earliestPutTs=1681844159075 2023-04-18 18:56:00,194 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting a3cf1b44f9bb4738bdd4ed715be29c70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1006, earliestPutTs=1681844159075 2023-04-18 18:56:00,195 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 49e4709b8d3743a7a26dd4afd5fc3fdb, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1028, earliestPutTs=1681844159202 2023-04-18 18:56:00,195 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 8e80d1fa52e64dc5b6a6fd0e502bda39, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1028, earliestPutTs=1681844159210 2023-04-18 18:56:00,211 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#B#compaction#495 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:00,214 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#A#compaction#496 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:00,215 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1396 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/f5f7d0c447b449e7a626f2e324a6c7eb 2023-04-18 18:56:00,218 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:00,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:00,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:00,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:00,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:00,228 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:00,228 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:00,228 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:00,228 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:00,228 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,228 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,230 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:00,230 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:00,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:00,231 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,231 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:00,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:00,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:00,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681844160220, max timestamp=1681844160232Num uniques 859; 2023-04-18 18:56:00,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681844160220, max timestamp=1681844160232Num uniques 859; 2023-04-18 18:56:00,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 18:56:00,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681844160220, max timestamp=1681844160232Num uniques 859; 2023-04-18 18:56:00,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 18:56:00,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 18:56:00,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:00,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:00,244 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:00,244 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:00,244 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:00,246 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:00,246 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:00,246 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:00,246 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:00,246 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:00,246 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:00,246 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:00,246 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:00,246 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,247 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:00,247 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:00,249 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,249 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:00,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:00,249 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,249 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:00,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:00,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:00,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:00,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:00,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64febb41[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:00,252 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:00,252 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,253 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,253 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:00,254 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:00,254 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:00,254 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,254 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:00,256 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:00,256 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,256 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:00,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:00,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:00,256 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:00,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:00,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:00,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:00,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:00,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:00,258 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:00,258 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:00,258 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:00,258 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:00,258 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:00,260 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,260 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:00,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:00,260 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:00,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:00,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:00,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:00,261 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,261 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/e98e46ed57204df6adf7210df10c21c5 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/e98e46ed57204df6adf7210df10c21c5 2023-04-18 18:56:00,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:00,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,262 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:00,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,266 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:00,266 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:00,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:00,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:00,266 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,266 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:00,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:00,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:00,266 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,266 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,266 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,266 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,266 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,266 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:00,266 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:00,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:00,266 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:00,266 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/A of 814b3c470b4969ca1c95de968f5dec66 into e98e46ed57204df6adf7210df10c21c5(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:00,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:00,266 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:00,266 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/A, priority=13, startTime=1681844160187; duration=0sec 2023-04-18 18:56:00,267 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:00,267 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:00,267 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:00,267 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/C is initiating minor compaction (all files) 2023-04-18 18:56:00,268 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/C in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:00,268 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/e2dc6cf456834b07a81aee271c88e8f1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/af648e86e37641009c4770bfdff8fc1d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/58b159a091924a68b4173c292fe9ed14] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=36.8 K 2023-04-18 18:56:00,268 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting e2dc6cf456834b07a81aee271c88e8f1, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=992, earliestPutTs=1681844158752 2023-04-18 18:56:00,268 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting af648e86e37641009c4770bfdff8fc1d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1006, earliestPutTs=1681844159075 2023-04-18 18:56:00,269 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 58b159a091924a68b4173c292fe9ed14, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1028, earliestPutTs=1681844159210 2023-04-18 18:56:00,283 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1396 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/e3a4f93c371d429dbdf4f32167d936cf 2023-04-18 18:56:00,284 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#C#compaction#498 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:00,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844220271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844220272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844220273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844220274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844220278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,301 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1396 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/55e300a57e4f4c4f9cc0a9a41403c776 2023-04-18 18:56:00,305 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/f5f7d0c447b449e7a626f2e324a6c7eb as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/f5f7d0c447b449e7a626f2e324a6c7eb 2023-04-18 18:56:00,309 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/f5f7d0c447b449e7a626f2e324a6c7eb, entries=400, sequenceid=1396, filesize=23.8 K 2023-04-18 18:56:00,309 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/e3a4f93c371d429dbdf4f32167d936cf as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/e3a4f93c371d429dbdf4f32167d936cf 2023-04-18 18:56:00,313 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/e3a4f93c371d429dbdf4f32167d936cf, entries=150, sequenceid=1396, filesize=11.9 K 2023-04-18 18:56:00,313 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/55e300a57e4f4c4f9cc0a9a41403c776 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/55e300a57e4f4c4f9cc0a9a41403c776 2023-04-18 18:56:00,323 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/55e300a57e4f4c4f9cc0a9a41403c776, entries=150, sequenceid=1396, filesize=11.9 K 2023-04-18 18:56:00,324 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~143.06 KB/146496, currentSize=134.18 KB/137400 for 814b3c470b4969ca1c95de968f5dec66 in 135ms, sequenceid=1396, compaction requested=false 2023-04-18 18:56:00,324 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:00,366 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:00,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:00,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:00,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:00,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:00,370 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:00,370 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:00,370 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:00,370 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:00,370 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,370 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,371 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:00,371 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:00,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:00,372 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,372 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:00,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:00,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:00,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:00,373 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:00,373 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:00,373 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:00,373 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:00,374 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:00,374 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:00,374 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:00,374 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-18 18:56:00,374 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:56:00,374 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:00,374 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:56:00,374 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:00,374 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:56:00,374 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:00,392 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1432 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/ca0c9dcbc05c4bd29c45239ec6b752a9 2023-04-18 18:56:00,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:56:00,396 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:00,402 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1432 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/8dbc73d7ebda4d78b2f46e8fa330224d 2023-04-18 18:56:00,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844220400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844220400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844220401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844220410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,422 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1432 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/9aeccab4eb7a4dddbf540a61ade98206 2023-04-18 18:56:00,426 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/ca0c9dcbc05c4bd29c45239ec6b752a9 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/ca0c9dcbc05c4bd29c45239ec6b752a9 2023-04-18 18:56:00,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844220415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,430 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/ca0c9dcbc05c4bd29c45239ec6b752a9, entries=150, sequenceid=1432, filesize=11.9 K 2023-04-18 18:56:00,431 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/8dbc73d7ebda4d78b2f46e8fa330224d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8dbc73d7ebda4d78b2f46e8fa330224d 2023-04-18 18:56:00,434 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8dbc73d7ebda4d78b2f46e8fa330224d, entries=150, sequenceid=1432, filesize=11.9 K 2023-04-18 18:56:00,434 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/9aeccab4eb7a4dddbf540a61ade98206 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/9aeccab4eb7a4dddbf540a61ade98206 2023-04-18 18:56:00,439 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/9aeccab4eb7a4dddbf540a61ade98206, entries=150, sequenceid=1432, filesize=11.9 K 2023-04-18 18:56:00,439 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 814b3c470b4969ca1c95de968f5dec66 in 65ms, sequenceid=1432, compaction requested=true 2023-04-18 18:56:00,440 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:00,440 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:00,440 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:00,440 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:00,440 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,440 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:00,440 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:00,443 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,443 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:00,444 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,444 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:00,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:00,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42a65a95[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:00,447 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:00,447 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,448 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:00,448 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,448 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:00,450 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:00,450 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:00,450 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:00,451 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:00,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:00,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:00,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:00,454 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:00,454 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:00,455 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:00,455 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:00,455 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:00,456 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,456 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:00,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:00,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:00,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,457 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:00,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:00,457 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,459 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:00,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,465 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,465 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,465 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,465 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:00,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:00,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:00,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:00,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:00,465 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:00,465 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:00,466 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:00,465 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,466 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,466 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,466 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,466 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:00,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:00,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:00,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:56:00,519 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-18 18:56:00,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:56:00,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:00,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:56:00,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:00,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:56:00,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:00,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844220555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844220556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,566 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:00,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:00,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:00,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844220557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844220557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844220560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:00,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:00,580 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:00,580 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:00,580 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:00,580 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:00,582 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,582 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,583 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:00,583 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,583 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,584 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:00,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:00,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:00,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:00,585 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:00,585 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:00,585 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:00,588 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:00,588 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:00,588 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:00,588 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:00,588 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:00,588 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:00,588 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:00,588 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:00,588 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,588 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:00,588 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:00,591 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:00,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:00,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30371bd0[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:00,595 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:00,595 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,599 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,599 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:00,599 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:00,599 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:00,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:00,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:00,600 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:00,601 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,601 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:00,601 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:00,601 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:00,601 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:00,601 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:00,601 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:00,601 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:00,603 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,603 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:00,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,603 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:00,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:00,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:00,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:00,604 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:00,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,605 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:00,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,609 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:00,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:00,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:00,610 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:00,610 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:00,610 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,610 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,610 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,610 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,610 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,610 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,610 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,611 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:00,611 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:00,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:00,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:00,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:00,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:00,650 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/04a1d1223fcf4c39945998f9738fe830 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/04a1d1223fcf4c39945998f9738fe830 2023-04-18 18:56:00,655 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/B of 814b3c470b4969ca1c95de968f5dec66 into 04a1d1223fcf4c39945998f9738fe830(size=13.1 K), total size for store is 36.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:00,655 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:00,655 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/B, priority=13, startTime=1681844160187; duration=0sec 2023-04-18 18:56:00,655 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:00,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844220661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844220665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844220671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 397 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844220673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844220674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,701 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/e8d5d33b38bf41ee8697653f5e73612d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/e8d5d33b38bf41ee8697653f5e73612d 2023-04-18 18:56:00,708 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/C of 814b3c470b4969ca1c95de968f5dec66 into e8d5d33b38bf41ee8697653f5e73612d(size=13.1 K), total size for store is 36.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:00,708 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:00,708 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/C, priority=13, startTime=1681844160187; duration=0sec 2023-04-18 18:56:00,709 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:00,710 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:00,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:00,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:00,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:00,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:00,714 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:00,714 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:00,714 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:00,714 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:00,715 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,715 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,717 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:00,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:00,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:00,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:00,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:00,718 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,719 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,719 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:00,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:00,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:00,724 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:00,724 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:00,724 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:00,724 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:00,724 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:00,724 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:00,724 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:00,724 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:00,724 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:00,725 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:00,725 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:00,725 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,725 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:00,725 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:00,726 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,727 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,727 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:00,727 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:00,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:00,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:00,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:00,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:00,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4255ba2d[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:00,730 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:00,730 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,732 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,732 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:00,732 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:00,732 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:00,732 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,732 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:00,733 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,733 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:00,733 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:00,733 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:00,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:00,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:00,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:00,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:00,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:00,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:00,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:00,738 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:00,738 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:00,738 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:00,739 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:00,739 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:00,743 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,743 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:00,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,743 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:00,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:00,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:00,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:00,744 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:00,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:00,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,745 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:00,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:00,748 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,748 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,748 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,748 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,748 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,748 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,748 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,749 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:00,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:00,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:00,749 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:00,749 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:00,749 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:00,749 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:00,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:00,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:00,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:00,849 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:00,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:00,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:00,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:00,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:00,853 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:00,853 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:00,853 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:00,853 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:00,854 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,854 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,856 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,856 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:00,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:00,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:00,856 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,856 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:00,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,857 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:00,857 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:00,857 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:00,857 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:00,857 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:00,857 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:00,858 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:00,858 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:00,858 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:00,858 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:00,858 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:00,858 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:00,858 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:00,858 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:00,858 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,858 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:00,858 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:00,859 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,860 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:00,861 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,861 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:00,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:00,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:00,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53603595[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:00,864 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:00,864 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,867 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,867 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:00,867 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:00,867 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:00,867 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,867 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:00,868 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,868 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:00,868 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:00,868 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:00,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:00,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:00,874 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:00,874 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:00,874 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:00,874 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:00,874 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:00,876 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,876 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:00,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,877 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:00,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:00,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:00,878 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:00,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,881 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:00,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844220874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844220874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844220877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 399 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844220877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,888 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:00,889 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:00,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:00,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:00,889 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,889 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,889 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,889 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,889 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,889 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:00,889 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,889 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:00,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:00,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:00,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:00,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:00,890 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:00,890 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:00,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:00,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844220885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:00,983 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1444 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/f2d541815fa2438e8e110e5255099db3 2023-04-18 18:56:00,990 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:00,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:00,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:00,994 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1444 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/d73cf1c67df34644be4d83d34f15834e 2023-04-18 18:56:00,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:00,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:00,996 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:00,996 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:00,996 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:00,996 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:00,997 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:00,997 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,998 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:00,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:00,999 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,000 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,000 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:01,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:01,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:01,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:01,003 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:01,003 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:01,003 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:01,004 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:01,004 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:01,004 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:01,004 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:01,004 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:01,004 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:01,004 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:01,004 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:01,004 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,004 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:01,004 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:01,006 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,006 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,006 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,006 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:01,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2362af9e[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:01,008 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:01,008 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,010 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1444 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/5dc07cad03974bd79c494cc237b1a985 2023-04-18 18:56:01,014 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/f2d541815fa2438e8e110e5255099db3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/f2d541815fa2438e8e110e5255099db3 2023-04-18 18:56:01,014 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,015 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:01,015 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:01,015 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:01,015 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,015 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:01,016 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,016 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:01,016 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:01,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,016 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:01,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,017 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/f2d541815fa2438e8e110e5255099db3, entries=200, sequenceid=1444, filesize=14.3 K 2023-04-18 18:56:01,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,018 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/d73cf1c67df34644be4d83d34f15834e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/d73cf1c67df34644be4d83d34f15834e 2023-04-18 18:56:01,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:01,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:01,020 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:01,020 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:01,020 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:01,020 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:01,020 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:01,021 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,021 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,021 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:01,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:01,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:01,022 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/d73cf1c67df34644be4d83d34f15834e, entries=100, sequenceid=1444, filesize=9.5 K 2023-04-18 18:56:01,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,023 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/5dc07cad03974bd79c494cc237b1a985 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/5dc07cad03974bd79c494cc237b1a985 2023-04-18 18:56:01,023 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,025 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,027 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/5dc07cad03974bd79c494cc237b1a985, entries=100, sequenceid=1444, filesize=9.5 K 2023-04-18 18:56:01,027 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:01,027 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:01,027 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:01,027 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:01,028 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,028 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 814b3c470b4969ca1c95de968f5dec66 in 509ms, sequenceid=1444, compaction requested=true 2023-04-18 18:56:01,028 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:01,027 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:01,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:01,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:01,028 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:01,028 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:01,028 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,028 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:01,028 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:01,028 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,028 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:56:01,028 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:01,028 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:01,028 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:56:01,028 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:01,029 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 64610 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:56:01,029 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/A is initiating minor compaction (all files) 2023-04-18 18:56:01,029 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47526 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:56:01,029 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/A in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:01,029 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/B is initiating minor compaction (all files) 2023-04-18 18:56:01,030 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/e98e46ed57204df6adf7210df10c21c5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/f5f7d0c447b449e7a626f2e324a6c7eb, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/ca0c9dcbc05c4bd29c45239ec6b752a9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/f2d541815fa2438e8e110e5255099db3] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=63.1 K 2023-04-18 18:56:01,030 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/B in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:01,030 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/04a1d1223fcf4c39945998f9738fe830, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/e3a4f93c371d429dbdf4f32167d936cf, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8dbc73d7ebda4d78b2f46e8fa330224d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/d73cf1c67df34644be4d83d34f15834e] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=46.4 K 2023-04-18 18:56:01,030 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting e98e46ed57204df6adf7210df10c21c5, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=1028, earliestPutTs=1681844159210 2023-04-18 18:56:01,030 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 04a1d1223fcf4c39945998f9738fe830, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=1028, earliestPutTs=1681844159210 2023-04-18 18:56:01,030 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting f5f7d0c447b449e7a626f2e324a6c7eb, keycount=400, bloomtype=ROW, size=23.8 K, encoding=NONE, compression=NONE, seqNum=1396, earliestPutTs=1681844160159 2023-04-18 18:56:01,030 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting e3a4f93c371d429dbdf4f32167d936cf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1396, earliestPutTs=1681844160159 2023-04-18 18:56:01,030 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting ca0c9dcbc05c4bd29c45239ec6b752a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1432, earliestPutTs=1681844160271 2023-04-18 18:56:01,030 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 8dbc73d7ebda4d78b2f46e8fa330224d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1432, earliestPutTs=1681844160271 2023-04-18 18:56:01,031 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting f2d541815fa2438e8e110e5255099db3, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1444, earliestPutTs=1681844160407 2023-04-18 18:56:01,031 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting d73cf1c67df34644be4d83d34f15834e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1444, earliestPutTs=1681844160409 2023-04-18 18:56:01,039 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#B#compaction#507 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:01,039 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#A#compaction#506 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:01,048 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/9b0986a8e8194b6a8ed9d9f84770a987 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/9b0986a8e8194b6a8ed9d9f84770a987 2023-04-18 18:56:01,050 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/6f3c3426817b4b63a6c7c6426785448a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/6f3c3426817b4b63a6c7c6426785448a 2023-04-18 18:56:01,053 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/B of 814b3c470b4969ca1c95de968f5dec66 into 9b0986a8e8194b6a8ed9d9f84770a987(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:01,053 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:01,053 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/B, priority=12, startTime=1681844161028; duration=0sec 2023-04-18 18:56:01,053 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:01,054 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:56:01,055 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47526 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:56:01,055 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/C is initiating minor compaction (all files) 2023-04-18 18:56:01,055 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/C in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:01,055 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/e8d5d33b38bf41ee8697653f5e73612d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/55e300a57e4f4c4f9cc0a9a41403c776, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/9aeccab4eb7a4dddbf540a61ade98206, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/5dc07cad03974bd79c494cc237b1a985] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=46.4 K 2023-04-18 18:56:01,055 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/A of 814b3c470b4969ca1c95de968f5dec66 into 6f3c3426817b4b63a6c7c6426785448a(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:01,055 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting e8d5d33b38bf41ee8697653f5e73612d, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=1028, earliestPutTs=1681844159210 2023-04-18 18:56:01,055 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:01,055 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/A, priority=12, startTime=1681844161028; duration=0sec 2023-04-18 18:56:01,056 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:01,056 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 55e300a57e4f4c4f9cc0a9a41403c776, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1396, earliestPutTs=1681844160159 2023-04-18 18:56:01,056 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 9aeccab4eb7a4dddbf540a61ade98206, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1432, earliestPutTs=1681844160271 2023-04-18 18:56:01,056 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 5dc07cad03974bd79c494cc237b1a985, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1444, earliestPutTs=1681844160409 2023-04-18 18:56:01,063 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#C#compaction#508 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:01,128 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:01,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:01,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:01,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:01,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:01,132 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:01,132 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:01,132 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:01,132 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:01,132 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,132 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,135 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:01,135 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:01,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:01,135 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,136 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:01,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:01,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:01,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:01,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:01,137 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:01,137 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:01,137 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:01,137 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:01,137 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:01,137 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:01,137 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:01,137 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 18:56:01,138 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:56:01,138 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:01,138 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:56:01,138 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:01,138 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:56:01,138 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:01,147 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1468 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/8dda2be568f74912bafbf78a1578907b 2023-04-18 18:56:01,162 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1468 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/303d3b16cbab45718c2afd33fcadf332 2023-04-18 18:56:01,173 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1468 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/85eada4a65b44346aca6266fc56894e5 2023-04-18 18:56:01,177 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/8dda2be568f74912bafbf78a1578907b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/8dda2be568f74912bafbf78a1578907b 2023-04-18 18:56:01,181 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/8dda2be568f74912bafbf78a1578907b, entries=150, sequenceid=1468, filesize=11.9 K 2023-04-18 18:56:01,182 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/303d3b16cbab45718c2afd33fcadf332 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/303d3b16cbab45718c2afd33fcadf332 2023-04-18 18:56:01,186 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/303d3b16cbab45718c2afd33fcadf332, entries=150, sequenceid=1468, filesize=11.9 K 2023-04-18 18:56:01,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:56:01,187 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:01,188 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/85eada4a65b44346aca6266fc56894e5 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/85eada4a65b44346aca6266fc56894e5 2023-04-18 18:56:01,191 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/85eada4a65b44346aca6266fc56894e5, entries=150, sequenceid=1468, filesize=11.9 K 2023-04-18 18:56:01,192 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=20.13 KB/20610 for 814b3c470b4969ca1c95de968f5dec66 in 55ms, sequenceid=1468, compaction requested=false 2023-04-18 18:56:01,192 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:01,192 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:01,192 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:01,192 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:01,192 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,192 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:01,192 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:01,194 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,194 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,194 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,194 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:01,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:01,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b03c9b9[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:01,196 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:01,196 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,197 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,197 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:01,197 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:01,197 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:01,197 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,197 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:01,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:56:01,198 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 18:56:01,199 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:56:01,199 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:01,199 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:56:01,199 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:01,199 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:56:01,199 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:01,199 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,199 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:01,199 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:01,199 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:01,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:01,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:01,203 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:01,203 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:01,203 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:01,203 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:01,203 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:01,206 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,206 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,206 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:01,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:01,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:01,207 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,208 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,211 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,211 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,211 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:01,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:01,211 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:01,212 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:01,211 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:01,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:01,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:01,211 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,212 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,212 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,212 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,212 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:01,212 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:01,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:01,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:01,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:01,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681844161217, max timestamp=1681844161233Num uniques 859; 2023-04-18 18:56:01,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681844161217, max timestamp=1681844161233Num uniques 859; 2023-04-18 18:56:01,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681844161217, max timestamp=1681844161233Num uniques 859; 2023-04-18 18:56:01,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 18:56:01,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 18:56:01,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 18:56:01,265 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:01,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844221262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:01,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 416 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844221263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:01,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 409 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844221264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:01,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844221264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:01,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 411 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844221265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,312 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:01,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:01,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:01,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:01,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:01,315 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:01,315 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:01,315 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:01,315 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:01,316 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,316 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,317 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:01,317 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:01,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:01,317 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,317 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:01,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:01,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:01,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:01,318 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:01,318 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:01,318 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:01,318 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:01,319 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:01,319 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:01,319 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:01,319 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:01,319 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:01,319 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:01,319 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:01,319 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,319 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:01,319 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:01,320 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,320 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,321 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,321 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:01,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:01,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44e89a26[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:01,322 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:01,322 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,324 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,324 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:01,324 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:01,324 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:01,324 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,324 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:01,325 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,325 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:01,325 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:01,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,325 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:01,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:01,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:01,328 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:01,328 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:01,328 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:01,328 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:01,328 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:01,330 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,330 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,330 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:01,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,330 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:01,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:01,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,333 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,335 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,335 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,335 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,335 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,335 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:01,335 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:01,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:01,335 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:01,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:01,335 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:01,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:01,335 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:01,335 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:01,335 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:01,335 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:01,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:01,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:01,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844221367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:01,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 418 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844221372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:01,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 411 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844221378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:01,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844221378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:01,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 413 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844221378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,436 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:01,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:01,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:01,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:01,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:01,439 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:01,439 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:01,439 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:01,439 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:01,440 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,440 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,442 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,442 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:01,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:01,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:01,443 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,443 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:01,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:01,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:01,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:01,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:01,444 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:01,444 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:01,444 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:01,444 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:01,444 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:01,445 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:01,445 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:01,445 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:01,445 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:01,445 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:01,445 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:01,445 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,445 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:01,445 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:01,449 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,449 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,449 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,449 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:01,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:01,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16f6c3b3[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:01,451 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:01,451 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,455 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,455 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:01,455 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,455 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:01,455 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:01,455 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:01,457 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:01,457 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:01,457 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,457 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:01,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:01,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:01,459 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:01,459 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:01,459 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:01,460 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:01,460 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:01,461 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,461 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,461 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:01,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:01,462 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,463 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,466 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,466 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:01,466 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:01,466 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,466 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,466 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:01,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:01,466 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:01,466 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,466 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,466 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:01,467 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:01,467 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:01,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:01,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:01,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:01,474 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/d5c94635be2d417c849122638d599ad9 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/d5c94635be2d417c849122638d599ad9 2023-04-18 18:56:01,478 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/C of 814b3c470b4969ca1c95de968f5dec66 into d5c94635be2d417c849122638d599ad9(size=13.2 K), total size for store is 25.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:01,478 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:01,478 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/C, priority=12, startTime=1681844161028; duration=0sec 2023-04-18 18:56:01,478 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:01,567 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:01,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:01,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:01,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:01,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:01,570 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:01,570 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:01,570 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:01,570 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:01,571 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,571 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,572 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:01,572 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:01,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:01,572 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,572 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:01,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:01,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:01,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:01,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:01,573 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:01,573 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:01,573 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:01,573 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:01,574 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:01,574 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:01,574 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:01,574 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:01,574 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:01,574 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:01,574 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:01,574 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,574 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:01,574 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:01,575 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,575 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,576 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,576 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:01,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:01,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@205e90ac[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:01,577 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:01,577 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,578 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:01,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844221570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,579 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,579 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:01,579 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:01,579 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:01,579 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:01,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:01,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 420 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844221576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,581 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,581 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:01,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,581 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:01,581 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:01,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:01,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:01,584 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:01,584 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:01,584 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:01,584 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:01,584 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:01,586 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,586 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,586 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:01,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:01,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:01,587 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,588 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,590 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,590 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,590 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,590 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,590 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,590 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,590 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:01,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:01,590 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:01,590 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:01,590 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:01,590 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:01,590 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:01,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:01,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:01,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:01,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:01,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:01,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:01,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 413 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844221584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:01,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844221585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:01,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 415 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844221585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,625 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1478 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/b8f99df816764e669d6afeedc52cd821 2023-04-18 18:56:01,634 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1478 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/6ca925517b5c44339371661b9aa96195 2023-04-18 18:56:01,690 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:01,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:01,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:01,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:01,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:01,694 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:01,694 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:01,694 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:01,694 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:01,695 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,695 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,696 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:01,696 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:01,696 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,697 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:01,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:01,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:01,698 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:01,698 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:01,698 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:01,698 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:01,698 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:01,698 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:01,698 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:01,698 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:01,698 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:01,698 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:01,698 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:01,698 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:01,698 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:01,698 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,698 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:01,698 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:01,703 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,703 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,703 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,703 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:01,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:01,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1525db1f[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:01,705 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:01,705 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,706 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,706 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:01,706 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:01,706 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:01,706 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,706 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:01,707 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:01,707 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:01,708 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:01,708 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:01,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:01,710 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:01,710 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:01,710 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:01,710 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:01,710 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:01,712 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,712 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,712 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:01,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,712 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,713 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:01,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,716 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,716 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,716 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:01,716 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:01,717 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:01,716 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:01,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:01,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:01,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:01,716 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,717 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,717 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,717 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,717 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:01,717 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:01,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:01,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:01,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:01,817 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:01,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:01,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:01,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:01,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:01,820 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:01,820 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:01,821 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:01,821 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:01,821 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,821 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,823 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:01,823 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:01,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:01,823 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,823 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:01,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:01,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:01,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:01,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:01,824 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:01,824 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:01,824 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:01,825 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:01,825 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:01,825 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:01,825 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:01,825 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:01,825 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:01,825 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:01,825 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:01,825 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,825 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:01,825 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:01,826 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,826 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,827 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,827 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:01,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@151c4a37[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:01,828 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:01,828 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,830 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,830 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,831 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:01,830 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:01,831 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:01,831 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:01,832 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:01,832 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:01,832 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:01,832 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:01,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:01,835 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:01,835 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:01,835 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:01,835 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:01,835 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:01,836 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,836 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,837 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:01,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:01,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:01,837 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,838 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,840 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,840 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,840 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:01,840 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,841 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:01,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:01,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:01,841 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:01,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:01,840 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:01,841 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:01,841 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:01,841 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,841 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:01,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:01,841 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:01,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844221881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:01,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 422 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844221883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:01,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 408 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844221894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:01,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 415 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844221895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:01,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 417 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844221896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,941 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:01,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:01,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:01,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:01,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:01,945 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:01,945 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:01,945 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:01,945 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:01,945 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,945 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,947 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,947 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:01,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:01,947 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,948 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:01,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:01,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:01,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:01,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:01,949 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:01,949 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:01,949 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:01,949 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:01,949 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:01,949 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:01,949 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:01,949 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:01,949 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:01,949 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:01,949 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:01,950 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,950 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:01,950 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:01,952 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,952 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,952 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,952 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:01,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:01,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35d7f87c[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:01,955 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:01,955 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,956 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,957 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,957 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:01,957 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:01,957 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:01,957 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:01,958 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:01,958 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:01,958 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,958 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:01,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:01,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:01,961 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:01,961 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:01,961 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:01,961 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:01,961 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:01,965 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,965 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:01,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,965 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:01,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:01,966 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:01,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:01,967 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,969 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:01,969 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:01,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:01,969 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,969 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:01,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:01,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:01,970 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:01,970 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:01,970 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,970 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:01,970 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:01,970 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:01,970 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:01,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:01,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:01,970 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:02,042 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1478 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/a2953d0b4315499285084494db172d9d 2023-04-18 18:56:02,047 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/b8f99df816764e669d6afeedc52cd821 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b8f99df816764e669d6afeedc52cd821 2023-04-18 18:56:02,050 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b8f99df816764e669d6afeedc52cd821, entries=250, sequenceid=1478, filesize=16.7 K 2023-04-18 18:56:02,051 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/6ca925517b5c44339371661b9aa96195 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/6ca925517b5c44339371661b9aa96195 2023-04-18 18:56:02,054 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/6ca925517b5c44339371661b9aa96195, entries=100, sequenceid=1478, filesize=9.5 K 2023-04-18 18:56:02,055 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/a2953d0b4315499285084494db172d9d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/a2953d0b4315499285084494db172d9d 2023-04-18 18:56:02,058 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/a2953d0b4315499285084494db172d9d, entries=100, sequenceid=1478, filesize=9.5 K 2023-04-18 18:56:02,058 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 814b3c470b4969ca1c95de968f5dec66 in 860ms, sequenceid=1478, compaction requested=true 2023-04-18 18:56:02,058 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:02,058 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:02,059 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:02,059 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:02,059 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:02,059 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35476 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:02,059 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:02,059 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/A is initiating minor compaction (all files) 2023-04-18 18:56:02,060 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/A in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:02,060 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:02,060 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/6f3c3426817b4b63a6c7c6426785448a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/8dda2be568f74912bafbf78a1578907b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b8f99df816764e669d6afeedc52cd821] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=41.8 K 2023-04-18 18:56:02,060 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 6f3c3426817b4b63a6c7c6426785448a, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=1444, earliestPutTs=1681844160277 2023-04-18 18:56:02,061 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 8dda2be568f74912bafbf78a1578907b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1468, earliestPutTs=1681844160541 2023-04-18 18:56:02,061 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/B is initiating minor compaction (all files) 2023-04-18 18:56:02,061 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/B in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:02,061 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/9b0986a8e8194b6a8ed9d9f84770a987, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/303d3b16cbab45718c2afd33fcadf332, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/6ca925517b5c44339371661b9aa96195] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=34.6 K 2023-04-18 18:56:02,061 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting b8f99df816764e669d6afeedc52cd821, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=1478, earliestPutTs=1681844161190 2023-04-18 18:56:02,061 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 9b0986a8e8194b6a8ed9d9f84770a987, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=1444, earliestPutTs=1681844160277 2023-04-18 18:56:02,062 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 303d3b16cbab45718c2afd33fcadf332, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1468, earliestPutTs=1681844160541 2023-04-18 18:56:02,062 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 6ca925517b5c44339371661b9aa96195, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1478, earliestPutTs=1681844161195 2023-04-18 18:56:02,066 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#A#compaction#515 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:02,067 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#B#compaction#516 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:02,070 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:02,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:02,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:02,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:02,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:02,078 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:02,078 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:02,078 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:02,078 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:02,079 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:02,079 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:02,081 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:02,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:02,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:02,081 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,081 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:02,082 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:02,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:02,082 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:02,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:02,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:02,083 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:02,083 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:02,083 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:02,083 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:02,083 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:02,083 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:02,083 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:02,083 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-18 18:56:02,084 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:56:02,084 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:02,084 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:56:02,084 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:02,084 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:56:02,084 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:02,084 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/b7b4a6c2140542ae954645b7f3d76bda as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b7b4a6c2140542ae954645b7f3d76bda 2023-04-18 18:56:02,088 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/A of 814b3c470b4969ca1c95de968f5dec66 into b7b4a6c2140542ae954645b7f3d76bda(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:02,088 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:02,088 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/A, priority=13, startTime=1681844162058; duration=0sec 2023-04-18 18:56:02,088 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:02,089 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:02,090 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35476 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:02,090 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/C is initiating minor compaction (all files) 2023-04-18 18:56:02,090 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/C in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:02,090 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/d5c94635be2d417c849122638d599ad9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/85eada4a65b44346aca6266fc56894e5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/a2953d0b4315499285084494db172d9d] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=34.6 K 2023-04-18 18:56:02,091 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting d5c94635be2d417c849122638d599ad9, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=1444, earliestPutTs=1681844160277 2023-04-18 18:56:02,091 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 85eada4a65b44346aca6266fc56894e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1468, earliestPutTs=1681844160541 2023-04-18 18:56:02,092 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting a2953d0b4315499285084494db172d9d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1478, earliestPutTs=1681844161195 2023-04-18 18:56:02,097 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#C#compaction#518 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:02,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:56:02,388 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:02,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:02,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 399 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844222400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:02,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 410 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844222401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:02,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 427 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844222402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:02,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 419 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844222403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:02,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 417 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844222404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,485 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/18edebbb25e44287bbb647fdfa11a781 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/18edebbb25e44287bbb647fdfa11a781 2023-04-18 18:56:02,491 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/B of 814b3c470b4969ca1c95de968f5dec66 into 18edebbb25e44287bbb647fdfa11a781(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:02,491 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:02,491 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/B, priority=13, startTime=1681844162059; duration=0sec 2023-04-18 18:56:02,491 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:02,499 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1514 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/a35e47e492024befbb6e7a3b341bc01e 2023-04-18 18:56:02,508 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/f621c8b4b1cc463988b81aa920680df0 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f621c8b4b1cc463988b81aa920680df0 2023-04-18 18:56:02,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:02,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 401 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844222507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:02,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 429 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844222508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,515 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/C of 814b3c470b4969ca1c95de968f5dec66 into f621c8b4b1cc463988b81aa920680df0(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:02,515 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:02,515 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/C, priority=13, startTime=1681844162060; duration=0sec 2023-04-18 18:56:02,515 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:02,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:02,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 403 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844222712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:02,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 431 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844222715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,924 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1514 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/ef3af24d1809403ea6227b30ecb63cd4 2023-04-18 18:56:02,942 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1514 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/d1dfcc4110204369b0eecca204cdf3e2 2023-04-18 18:56:02,946 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/a35e47e492024befbb6e7a3b341bc01e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/a35e47e492024befbb6e7a3b341bc01e 2023-04-18 18:56:02,949 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/a35e47e492024befbb6e7a3b341bc01e, entries=150, sequenceid=1514, filesize=11.9 K 2023-04-18 18:56:02,949 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/ef3af24d1809403ea6227b30ecb63cd4 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ef3af24d1809403ea6227b30ecb63cd4 2023-04-18 18:56:02,953 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ef3af24d1809403ea6227b30ecb63cd4, entries=150, sequenceid=1514, filesize=11.9 K 2023-04-18 18:56:02,953 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/d1dfcc4110204369b0eecca204cdf3e2 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/d1dfcc4110204369b0eecca204cdf3e2 2023-04-18 18:56:02,956 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/d1dfcc4110204369b0eecca204cdf3e2, entries=150, sequenceid=1514, filesize=11.9 K 2023-04-18 18:56:02,957 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 814b3c470b4969ca1c95de968f5dec66 in 874ms, sequenceid=1514, compaction requested=false 2023-04-18 18:56:02,957 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:02,957 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:02,957 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:02,957 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:02,957 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,957 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:02,957 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:02,959 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,959 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:02,960 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:02,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:02,960 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:02,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:02,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:02,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:02,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61d8d59a[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:02,961 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:02,961 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:02,963 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:02,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:02,963 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:02,963 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:02,963 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:02,963 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,963 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:02,964 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,964 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:02,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:02,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:02,965 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:02,965 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:02,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:02,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:02,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:02,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:02,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:02,967 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:02,968 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:02,968 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:02,968 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:02,968 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:02,970 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:02,970 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:02,970 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:02,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:02,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:02,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:02,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:02,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:02,970 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:02,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:02,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:02,971 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:02,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:02,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,973 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,974 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:02,974 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:02,974 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:02,974 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:02,974 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:02,974 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:02,974 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:02,974 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:02,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:02,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:02,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:02,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:02,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:02,974 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:02,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:02,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:02,975 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:02,975 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:03,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:56:03,024 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-18 18:56:03,024 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:56:03,024 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:03,024 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:56:03,024 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:03,024 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:56:03,024 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:03,032 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1528 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/3264b87f8868468e93d03835bce39ac5 2023-04-18 18:56:03,042 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1528 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/7af60467bbfe4928b7cd141076467d3a 2023-04-18 18:56:03,050 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1528 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/adc5017262ea40788132865fcebad743 2023-04-18 18:56:03,054 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/3264b87f8868468e93d03835bce39ac5 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/3264b87f8868468e93d03835bce39ac5 2023-04-18 18:56:03,057 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/3264b87f8868468e93d03835bce39ac5, entries=150, sequenceid=1528, filesize=11.9 K 2023-04-18 18:56:03,058 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/7af60467bbfe4928b7cd141076467d3a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/7af60467bbfe4928b7cd141076467d3a 2023-04-18 18:56:03,063 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/7af60467bbfe4928b7cd141076467d3a, entries=100, sequenceid=1528, filesize=9.5 K 2023-04-18 18:56:03,063 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/adc5017262ea40788132865fcebad743 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/adc5017262ea40788132865fcebad743 2023-04-18 18:56:03,066 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/adc5017262ea40788132865fcebad743, entries=100, sequenceid=1528, filesize=9.5 K 2023-04-18 18:56:03,067 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=107.34 KB/109920 for 814b3c470b4969ca1c95de968f5dec66 in 43ms, sequenceid=1528, compaction requested=true 2023-04-18 18:56:03,067 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:03,067 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:03,067 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:03,067 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:03,067 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:03,067 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:03,068 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35578 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:03,068 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38018 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:03,068 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/B is initiating minor compaction (all files) 2023-04-18 18:56:03,068 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/A is initiating minor compaction (all files) 2023-04-18 18:56:03,068 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/B in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:03,068 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/18edebbb25e44287bbb647fdfa11a781, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ef3af24d1809403ea6227b30ecb63cd4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/7af60467bbfe4928b7cd141076467d3a] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=34.7 K 2023-04-18 18:56:03,068 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/A in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:03,068 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b7b4a6c2140542ae954645b7f3d76bda, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/a35e47e492024befbb6e7a3b341bc01e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/3264b87f8868468e93d03835bce39ac5] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=37.1 K 2023-04-18 18:56:03,070 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 18edebbb25e44287bbb647fdfa11a781, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=1478, earliestPutTs=1681844160541 2023-04-18 18:56:03,070 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting b7b4a6c2140542ae954645b7f3d76bda, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=1478, earliestPutTs=1681844160541 2023-04-18 18:56:03,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:56:03,070 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting ef3af24d1809403ea6227b30ecb63cd4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1514, earliestPutTs=1681844161263 2023-04-18 18:56:03,070 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting a35e47e492024befbb6e7a3b341bc01e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1514, earliestPutTs=1681844161263 2023-04-18 18:56:03,070 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 18:56:03,071 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:56:03,071 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 7af60467bbfe4928b7cd141076467d3a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1528, earliestPutTs=1681844162401 2023-04-18 18:56:03,071 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:03,071 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:56:03,071 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 3264b87f8868468e93d03835bce39ac5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1528, earliestPutTs=1681844162399 2023-04-18 18:56:03,071 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:03,071 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:56:03,071 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:03,075 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:03,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:03,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:03,080 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#B#compaction#525 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:03,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:03,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:03,080 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:03,080 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:03,080 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:03,080 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:03,081 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,081 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,083 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:03,083 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:03,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:03,083 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,083 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:03,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:03,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:03,085 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#A#compaction#526 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:03,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:03,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:03,096 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1548 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/b44d3604ff77438dba97a899fa35e92c 2023-04-18 18:56:03,100 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:03,100 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:03,100 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:03,101 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:03,101 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:03,101 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:03,101 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:03,101 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:03,101 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,101 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:03,101 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:03,103 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,103 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,103 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:03,107 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:03,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:03,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:03,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:03,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:03,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 446 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844223101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:03,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:03,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 419 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844223102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,110 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:03,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a1e179d[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:03,110 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,111 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,111 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:03,111 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:03,111 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:03,111 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,111 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:03,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:03,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:03,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:03,113 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:03,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:03,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:03,116 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:03,116 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:03,116 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:03,116 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:03,116 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:03,124 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,124 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:03,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,124 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:03,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:03,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:03,124 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:03,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:03,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:03,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,126 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:03,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,130 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,130 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,130 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,130 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:03,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:03,130 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:03,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:03,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:03,130 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,130 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,130 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,130 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:03,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:03,130 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:03,131 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:03,130 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:03,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:03,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:03,134 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1548 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/ed29e20f8fee4d3392f5e7c41f336c02 2023-04-18 18:56:03,136 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/960bdb439e5747c4be87d262d4ce8964 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/960bdb439e5747c4be87d262d4ce8964 2023-04-18 18:56:03,137 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/34e2fee7fc0741d2aebe5278fa832b79 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/34e2fee7fc0741d2aebe5278fa832b79 2023-04-18 18:56:03,142 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/A of 814b3c470b4969ca1c95de968f5dec66 into 960bdb439e5747c4be87d262d4ce8964(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:03,142 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:03,142 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/A, priority=13, startTime=1681844163067; duration=0sec 2023-04-18 18:56:03,142 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:03,142 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:03,143 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35578 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:03,143 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/C is initiating minor compaction (all files) 2023-04-18 18:56:03,143 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/C in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:03,143 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f621c8b4b1cc463988b81aa920680df0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/d1dfcc4110204369b0eecca204cdf3e2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/adc5017262ea40788132865fcebad743] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=34.7 K 2023-04-18 18:56:03,144 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting f621c8b4b1cc463988b81aa920680df0, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=1478, earliestPutTs=1681844160541 2023-04-18 18:56:03,144 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting d1dfcc4110204369b0eecca204cdf3e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1514, earliestPutTs=1681844161263 2023-04-18 18:56:03,145 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting adc5017262ea40788132865fcebad743, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1528, earliestPutTs=1681844162401 2023-04-18 18:56:03,146 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/B of 814b3c470b4969ca1c95de968f5dec66 into 34e2fee7fc0741d2aebe5278fa832b79(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:03,146 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:03,146 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/B, priority=13, startTime=1681844163067; duration=0sec 2023-04-18 18:56:03,146 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:03,153 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#C#compaction#529 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:03,155 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1548 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/bb063c7340f04a9db0d82c4735ba81be 2023-04-18 18:56:03,160 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/b44d3604ff77438dba97a899fa35e92c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b44d3604ff77438dba97a899fa35e92c 2023-04-18 18:56:03,164 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b44d3604ff77438dba97a899fa35e92c, entries=250, sequenceid=1548, filesize=16.7 K 2023-04-18 18:56:03,165 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/ed29e20f8fee4d3392f5e7c41f336c02 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ed29e20f8fee4d3392f5e7c41f336c02 2023-04-18 18:56:03,168 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ed29e20f8fee4d3392f5e7c41f336c02, entries=150, sequenceid=1548, filesize=11.9 K 2023-04-18 18:56:03,169 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/bb063c7340f04a9db0d82c4735ba81be as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/bb063c7340f04a9db0d82c4735ba81be 2023-04-18 18:56:03,173 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/bb063c7340f04a9db0d82c4735ba81be, entries=150, sequenceid=1548, filesize=11.9 K 2023-04-18 18:56:03,174 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 814b3c470b4969ca1c95de968f5dec66 in 104ms, sequenceid=1548, compaction requested=false 2023-04-18 18:56:03,174 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:03,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:56:03,213 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 18:56:03,213 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:56:03,213 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:03,214 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:56:03,214 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:03,214 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:56:03,214 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:03,221 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1563 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/4fa9033f473c489dbabbae8ee669adcf 2023-04-18 18:56:03,232 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:03,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:03,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:03,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:03,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:03,235 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:03,235 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:03,236 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:03,236 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:03,236 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,236 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,237 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:03,237 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,238 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,238 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:03,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:03,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:03,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:03,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:03,239 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:03,239 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:03,239 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:03,239 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:03,239 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:03,239 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:03,239 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:03,240 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:03,240 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:03,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:03,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:03,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:03,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:03,241 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:03,241 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:03,242 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,242 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:03,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:03,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2da4b31b[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:03,243 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:03,243 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,245 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,245 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:03,246 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:03,246 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:03,246 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,246 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:03,247 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:03,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:03,247 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:03,247 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:03,247 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:03,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:03,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:03,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:03,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:03,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:03,250 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:03,250 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:03,250 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:03,250 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:03,250 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:03,252 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,252 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,252 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:03,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:03,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:03,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,253 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,254 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,256 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:03,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:03,256 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,256 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:03,257 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:03,257 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:03,257 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:03,256 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:03,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:03,257 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:03,257 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,257 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,257 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,257 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:03,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:03,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 429 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844223257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:03,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 457 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844223265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,357 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:03,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:03,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:03,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:03,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:03,361 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:03,361 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:03,361 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:03,361 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:03,361 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,361 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,364 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:03,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:03,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:03,364 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,364 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,364 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:03,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:03,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:03,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:03,366 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:03,366 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:03,366 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:03,366 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:03,366 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:03,366 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:03,366 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:03,366 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:03,366 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:03,366 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:03,366 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:03,366 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,366 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:03,366 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:03,368 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,368 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:03,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:03,368 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,368 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:03,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:03,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:03,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:03,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@273abacb[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:03,370 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:03,370 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,371 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,371 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:03,371 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,371 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:03,371 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:03,371 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:03,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:03,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 431 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844223366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,372 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:03,372 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:03,372 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:03,372 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:03,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:03,375 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:03,375 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:03,375 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:03,375 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:03,375 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:03,377 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,377 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,377 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:03,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:03,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:03,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,377 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:03,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,379 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:03,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 459 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844223375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:03,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,381 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,381 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:03,381 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,381 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:03,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:03,382 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:03,381 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,382 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,382 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,382 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,382 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:03,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:03,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:03,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:03,382 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:03,382 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:03,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:03,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:03,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:03,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 412 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60796 deadline: 1681844223410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:03,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 421 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60820 deadline: 1681844223411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:03,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 419 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60778 deadline: 1681844223416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,482 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:03,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:03,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:03,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:03,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:03,486 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:03,486 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:03,486 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:03,486 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:03,486 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,486 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,487 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:03,487 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:03,488 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,488 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:03,489 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:03,489 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:03,489 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:03,489 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:03,489 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:03,489 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:03,489 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:03,490 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:03,490 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:03,490 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:03,490 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:03,490 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:03,490 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:03,490 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,490 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:03,490 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:03,491 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,491 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:03,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:03,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69dfb3[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:03,493 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:03,493 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,495 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,495 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:03,495 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:03,495 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:03,495 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,496 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:03,497 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,497 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:03,497 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:03,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:03,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:03,497 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:03,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:03,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:03,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:03,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:03,499 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:03,499 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:03,500 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:03,500 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:03,500 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:03,501 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,501 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,501 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:03,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,502 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:03,504 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,510 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,510 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,510 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:03,510 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:03,510 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:03,510 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:03,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:03,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:03,510 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:03,511 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,511 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,511 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,511 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:03,511 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:03,562 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/2b777b45249c4d09942ffa48ebdd62f1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/2b777b45249c4d09942ffa48ebdd62f1 2023-04-18 18:56:03,567 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/C of 814b3c470b4969ca1c95de968f5dec66 into 2b777b45249c4d09942ffa48ebdd62f1(size=13.4 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:03,567 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:03,567 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/C, priority=13, startTime=1681844163067; duration=0sec 2023-04-18 18:56:03,567 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:03,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:03,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 433 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844223574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:03,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 461 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844223580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,611 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:03,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:03,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:03,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:03,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:03,614 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:03,614 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:03,615 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:03,615 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:03,615 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,615 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,616 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:03,616 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:03,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:03,617 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,617 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:03,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:03,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:03,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:03,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:03,618 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:03,618 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:03,618 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:03,618 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:03,618 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:03,618 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:03,618 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:03,618 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:03,618 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:03,619 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:03,619 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:03,619 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,619 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:03,619 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:03,620 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,620 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:03,621 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,621 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:03,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:03,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ae192dc[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:03,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:03,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,623 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,623 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:03,623 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:03,623 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:03,623 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,624 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:03,626 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:03,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:03,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:03,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:03,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:03,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:03,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:03,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:03,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:03,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:03,628 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:03,628 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:03,629 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:03,629 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:03,629 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:03,630 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,630 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:03,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:03,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,630 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:03,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:03,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:03,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:03,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,631 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:03,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,633 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1563 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/80e45f16b2084660a3f43f256f121c13 2023-04-18 18:56:03,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:03,634 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,636 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:03,636 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,636 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:03,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:03,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:03,636 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,636 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,636 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,636 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,637 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,637 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,637 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:03,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:03,636 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:03,637 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:03,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:03,737 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:03,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:03,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:03,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:03,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:03,742 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:03,742 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:03,742 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:03,742 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:03,743 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,743 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,744 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,744 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:03,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:03,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:03,744 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,744 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:03,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:03,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:03,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:03,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:03,746 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:03,746 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:03,746 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:03,746 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:03,746 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:03,746 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:03,746 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:03,746 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:03,746 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:03,746 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:03,746 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:03,746 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,746 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:03,746 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:03,749 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,749 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:03,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:03,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:03,749 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,749 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:03,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:03,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:03,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:03,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22d5d0f0[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:03,751 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:03,751 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,752 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,752 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:03,752 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:03,752 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:03,752 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,752 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:03,753 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:03,753 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:03,753 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:03,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:03,754 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:03,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:03,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:03,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:03,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:03,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:03,756 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:03,756 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:03,756 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:03,756 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:03,757 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:03,758 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,758 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:03,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,758 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:03,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:03,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:03,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:03,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:03,760 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:03,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,762 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,766 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:03,766 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:03,766 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:03,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:03,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:03,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:03,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:03,766 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:03,766 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,766 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:03,766 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:03,766 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,766 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,766 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,766 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,766 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:03,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:03,866 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:03,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:03,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:03,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:03,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:03,870 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:03,870 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:03,870 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:03,870 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:03,871 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,871 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,872 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:03,872 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:03,872 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,872 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:03,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:03,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:03,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:03,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:03,874 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:03,874 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:03,874 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:03,874 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:03,874 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:03,874 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:03,874 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:03,874 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:03,874 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:03,874 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:03,874 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:03,874 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,874 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:03,874 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:03,876 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,876 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:03,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:03,876 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,876 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:03,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:03,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:03,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:03,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:03,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@504bca83[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:03,878 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:03,878 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,879 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,879 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:03,880 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,880 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:03,880 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:03,880 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:03,884 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:03,885 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:03,885 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:03,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:03,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 435 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844223881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,885 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:03,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:03,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:03,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:03,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:03,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:03,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:03,888 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:03,888 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:03,888 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:03,888 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:03,888 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:03,890 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:03,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:03,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:03,891 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,891 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,891 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:03,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:03,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 463 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844223887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:03,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:03,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:03,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,892 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:03,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:03,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,898 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:03,898 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:03,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:03,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:03,898 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,899 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,899 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:03,899 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:03,899 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,899 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:03,899 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:03,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:03,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:03,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:03,899 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:03,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:03,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:03,900 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:03,900 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:04,000 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:04,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:04,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:04,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:04,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:04,003 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:04,004 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:04,004 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:04,004 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:04,004 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:04,004 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:04,006 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,006 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:04,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:04,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:04,007 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,007 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:04,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:04,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:04,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:04,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:04,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:04,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:04,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:04,012 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:04,013 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:04,013 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:04,013 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:04,013 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:04,013 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:04,013 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:04,013 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:04,013 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:04,013 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:04,013 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:04,013 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,013 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:04,013 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:04,016 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:04,016 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:04,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:04,017 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:04,017 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:04,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:04,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:04,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:04,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:04,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:04,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e11037e[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:04,018 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:04,018 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:04,019 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:04,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:04,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:04,020 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,020 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:04,021 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:04,021 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:04,021 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:04,022 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:04,022 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:04,022 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,022 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:04,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:04,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:04,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:04,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:04,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:04,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:04,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:04,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:04,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:04,026 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:04,027 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:04,027 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:04,027 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:04,027 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:04,029 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:04,029 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:04,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:04,029 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:04,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:04,030 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:04,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:04,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:04,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,032 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:04,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:04,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,034 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,034 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:04,034 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:04,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:04,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:04,034 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:04,034 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,035 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:04,035 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:04,035 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:04,035 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:04,035 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:04,035 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:04,035 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:04,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:04,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:04,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:04,046 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1563 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/2f1a8848c2254c7581c075c9a9f3fab5 2023-04-18 18:56:04,050 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/4fa9033f473c489dbabbae8ee669adcf as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/4fa9033f473c489dbabbae8ee669adcf 2023-04-18 18:56:04,055 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/4fa9033f473c489dbabbae8ee669adcf, entries=200, sequenceid=1563, filesize=14.3 K 2023-04-18 18:56:04,056 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/80e45f16b2084660a3f43f256f121c13 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/80e45f16b2084660a3f43f256f121c13 2023-04-18 18:56:04,059 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/80e45f16b2084660a3f43f256f121c13, entries=150, sequenceid=1563, filesize=11.9 K 2023-04-18 18:56:04,060 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/2f1a8848c2254c7581c075c9a9f3fab5 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/2f1a8848c2254c7581c075c9a9f3fab5 2023-04-18 18:56:04,063 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/2f1a8848c2254c7581c075c9a9f3fab5, entries=150, sequenceid=1563, filesize=11.9 K 2023-04-18 18:56:04,064 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 814b3c470b4969ca1c95de968f5dec66 in 851ms, sequenceid=1563, compaction requested=true 2023-04-18 18:56:04,064 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:04,064 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:04,064 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:04,064 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:04,064 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:04,064 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:04,065 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 45444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:04,065 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38124 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:04,065 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/A is initiating minor compaction (all files) 2023-04-18 18:56:04,065 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/B is initiating minor compaction (all files) 2023-04-18 18:56:04,065 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/A in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:04,065 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/B in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:04,065 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/960bdb439e5747c4be87d262d4ce8964, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b44d3604ff77438dba97a899fa35e92c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/4fa9033f473c489dbabbae8ee669adcf] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=44.4 K 2023-04-18 18:56:04,065 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/34e2fee7fc0741d2aebe5278fa832b79, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ed29e20f8fee4d3392f5e7c41f336c02, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/80e45f16b2084660a3f43f256f121c13] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=37.2 K 2023-04-18 18:56:04,065 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 960bdb439e5747c4be87d262d4ce8964, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=1528, earliestPutTs=1681844161264 2023-04-18 18:56:04,066 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 34e2fee7fc0741d2aebe5278fa832b79, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=1528, earliestPutTs=1681844161264 2023-04-18 18:56:04,066 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting b44d3604ff77438dba97a899fa35e92c, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=1548, earliestPutTs=1681844163057 2023-04-18 18:56:04,066 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting ed29e20f8fee4d3392f5e7c41f336c02, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1548, earliestPutTs=1681844163057 2023-04-18 18:56:04,066 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 4fa9033f473c489dbabbae8ee669adcf, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1563, earliestPutTs=1681844163091 2023-04-18 18:56:04,066 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 80e45f16b2084660a3f43f256f121c13, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1563, earliestPutTs=1681844163099 2023-04-18 18:56:04,077 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#A#compaction#533 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:04,077 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#B#compaction#534 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:04,102 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/ccb9d88614be4eaba914c3ebff62afa4 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ccb9d88614be4eaba914c3ebff62afa4 2023-04-18 18:56:04,108 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/B of 814b3c470b4969ca1c95de968f5dec66 into ccb9d88614be4eaba914c3ebff62afa4(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:04,108 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:04,108 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/B, priority=13, startTime=1681844164064; duration=0sec 2023-04-18 18:56:04,108 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:04,108 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:04,109 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38124 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:04,109 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): 814b3c470b4969ca1c95de968f5dec66/C is initiating minor compaction (all files) 2023-04-18 18:56:04,110 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 814b3c470b4969ca1c95de968f5dec66/C in TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:04,110 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/2b777b45249c4d09942ffa48ebdd62f1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/bb063c7340f04a9db0d82c4735ba81be, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/2f1a8848c2254c7581c075c9a9f3fab5] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp, totalSize=37.2 K 2023-04-18 18:56:04,110 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 2b777b45249c4d09942ffa48ebdd62f1, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=1528, earliestPutTs=1681844161264 2023-04-18 18:56:04,110 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting bb063c7340f04a9db0d82c4735ba81be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1548, earliestPutTs=1681844163057 2023-04-18 18:56:04,111 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 2f1a8848c2254c7581c075c9a9f3fab5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1563, earliestPutTs=1681844163099 2023-04-18 18:56:04,117 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): 814b3c470b4969ca1c95de968f5dec66#C#compaction#535 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:04,135 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:04,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:04,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:04,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:04,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:04,142 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:04,142 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:04,142 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:04,142 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:04,144 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:04,144 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:04,145 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:04,145 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:04,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:04,145 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,145 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:04,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:04,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:04,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:04,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:04,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:04,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:04,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:04,147 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:04,147 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:04,147 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:04,147 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:04,147 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:04,147 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:04,147 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. started... 2023-04-18 18:56:04,147 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-18 18:56:04,147 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:56:04,147 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:04,147 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:56:04,147 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:04,147 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:56:04,147 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:04,161 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1584 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/e7519d97d8384ed9b0d97eecc74399f7 2023-04-18 18:56:04,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:56:04,390 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. as already flushing 2023-04-18 18:56:04,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:04,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 468 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844224414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:04,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 444 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844224418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,424 DEBUG [Thread-2972] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24a917df to 127.0.0.1:65091 2023-04-18 18:56:04,424 DEBUG [Thread-2972] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:04,426 DEBUG [Thread-2978] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6871d7d8 to 127.0.0.1:65091 2023-04-18 18:56:04,426 DEBUG [Thread-2978] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:04,431 DEBUG [Thread-2974] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78fa3dee to 127.0.0.1:65091 2023-04-18 18:56:04,431 DEBUG [Thread-2974] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:04,431 DEBUG [Thread-2976] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36d2d826 to 127.0.0.1:65091 2023-04-18 18:56:04,431 DEBUG [Thread-2976] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:04,433 DEBUG [Thread-2970] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4cefb012 to 127.0.0.1:65091 2023-04-18 18:56:04,433 DEBUG [Thread-2970] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:04,503 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/81a9fb08d90942738a05d7fff7db429d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/81a9fb08d90942738a05d7fff7db429d 2023-04-18 18:56:04,506 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/A of 814b3c470b4969ca1c95de968f5dec66 into 81a9fb08d90942738a05d7fff7db429d(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:04,506 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:04,506 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/A, priority=13, startTime=1681844164064; duration=0sec 2023-04-18 18:56:04,506 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:04,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:04,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 470 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844224520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:04,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 446 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844224524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,531 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/31dd6e37b6b447698eb7e70e7561fc60 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/31dd6e37b6b447698eb7e70e7561fc60 2023-04-18 18:56:04,534 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 814b3c470b4969ca1c95de968f5dec66/C of 814b3c470b4969ca1c95de968f5dec66 into 31dd6e37b6b447698eb7e70e7561fc60(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:04,534 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:04,534 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66., storeName=814b3c470b4969ca1c95de968f5dec66/C, priority=13, startTime=1681844164064; duration=0sec 2023-04-18 18:56:04,534 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:04,574 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1584 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/8cc90d941fb1474eba0d091c55944271 2023-04-18 18:56:04,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:04,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 472 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60816 deadline: 1681844224723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:04,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 448 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60806 deadline: 1681844224725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,983 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1584 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/79e358889c064432833364c3e2fb3463 2023-04-18 18:56:04,987 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/e7519d97d8384ed9b0d97eecc74399f7 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/e7519d97d8384ed9b0d97eecc74399f7 2023-04-18 18:56:04,991 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/e7519d97d8384ed9b0d97eecc74399f7, entries=150, sequenceid=1584, filesize=11.9 K 2023-04-18 18:56:04,991 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/8cc90d941fb1474eba0d091c55944271 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8cc90d941fb1474eba0d091c55944271 2023-04-18 18:56:04,994 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8cc90d941fb1474eba0d091c55944271, entries=150, sequenceid=1584, filesize=11.9 K 2023-04-18 18:56:04,994 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/79e358889c064432833364c3e2fb3463 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/79e358889c064432833364c3e2fb3463 2023-04-18 18:56:04,997 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/79e358889c064432833364c3e2fb3463, entries=150, sequenceid=1584, filesize=11.9 K 2023-04-18 18:56:04,998 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 814b3c470b4969ca1c95de968f5dec66 in 851ms, sequenceid=1584, compaction requested=false 2023-04-18 18:56:04,998 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:04,998 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:04,998 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:04,998 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:04,998 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:04,998 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:04,998 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:05,000 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:05,000 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:05,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:05,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:05,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:05,001 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:05,001 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:05,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:05,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:05,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:05,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:05,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:05,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:05,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c00512[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:05,002 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:05,002 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:05,003 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:05,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:05,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:05,003 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:05,004 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:05,004 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:05,004 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:05,004 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:05,005 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:05,005 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:05,005 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:05,005 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:05,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:05,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:05,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:05,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:05,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:05,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:05,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:05,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:05,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:05,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:05,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:05,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:05,007 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:05,007 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:05,007 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:05,007 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:05,007 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:05,008 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:05,008 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:05,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:05,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:05,008 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:05,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:05,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:05,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:05,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:05,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:05,009 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:05,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:05,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:05,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:05,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:05,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:05,010 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:05,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:05,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:05,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:05,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:05,013 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:05,013 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:05,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:05,013 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:05,013 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:05,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:05,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:05,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:05,013 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:05,013 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:05,013 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:05,013 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:05,013 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:05,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:05,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:05,013 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:05,013 DEBUG [Thread-2969] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:05,013 DEBUG [Thread-2969] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:05,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:56:05,026 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-18 18:56:05,026 DEBUG [Thread-2959] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x357cb237 to 127.0.0.1:65091 2023-04-18 18:56:05,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:56:05,026 DEBUG [Thread-2959] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:05,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:05,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:56:05,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:05,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:56:05,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:05,028 DEBUG [Thread-2965] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x09b8ff30 to 127.0.0.1:65091 2023-04-18 18:56:05,028 DEBUG [Thread-2965] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:05,114 DEBUG [Thread-2969] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:05,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:05,430 DEBUG [Thread-2963] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32cd672d to 127.0.0.1:65091 2023-04-18 18:56:05,430 DEBUG [Thread-2963] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:05,431 DEBUG [Thread-2961] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4539ed0c to 127.0.0.1:65091 2023-04-18 18:56:05,431 DEBUG [Thread-2961] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:05,433 DEBUG [Thread-2967] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4c050db8 to 127.0.0.1:65091 2023-04-18 18:56:05,433 DEBUG [Thread-2967] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:05,433 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-18 18:56:05,433 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 319 2023-04-18 18:56:05,433 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 284 2023-04-18 18:56:05,433 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 282 2023-04-18 18:56:05,433 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 289 2023-04-18 18:56:05,433 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 273 2023-04-18 18:56:05,433 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-18 18:56:05,433 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-18 18:56:05,433 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(402): scanned 1571 2023-04-18 18:56:05,433 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(403): verified 4713 rows 2023-04-18 18:56:05,433 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(402): scanned 1578 2023-04-18 18:56:05,433 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(403): verified 4731 rows 2023-04-18 18:56:05,433 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(402): scanned 1563 2023-04-18 18:56:05,433 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(403): verified 4689 rows 2023-04-18 18:56:05,433 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(402): scanned 1583 2023-04-18 18:56:05,433 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(403): verified 4749 rows 2023-04-18 18:56:05,433 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(402): scanned 1576 2023-04-18 18:56:05,433 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(403): verified 4728 rows 2023-04-18 18:56:05,434 INFO [Listener at localhost/36951] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-18 18:56:05,434 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3fb41c50 to 127.0.0.1:65091 2023-04-18 18:56:05,434 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:05,435 INFO [Listener at localhost/36951] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-18 18:56:05,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-18 18:56:05,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-18 18:56:05,440 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1600 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/25d3ca8d0f45478db0d8abf15e8cc246 2023-04-18 18:56:05,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-18 18:56:05,440 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844165440"}]},"ts":"1681844165440"} 2023-04-18 18:56:05,441 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-18 18:56:05,443 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-18 18:56:05,443 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=814b3c470b4969ca1c95de968f5dec66, UNASSIGN}] 2023-04-18 18:56:05,445 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=814b3c470b4969ca1c95de968f5dec66, UNASSIGN 2023-04-18 18:56:05,445 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=814b3c470b4969ca1c95de968f5dec66, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:05,446 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844165445"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681844165445"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681844165445"}]},"ts":"1681844165445"} 2023-04-18 18:56:05,446 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952}] 2023-04-18 18:56:05,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-18 18:56:05,598 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:56:05,598 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 814b3c470b4969ca1c95de968f5dec66, disabling compactions & flushes 2023-04-18 18:56:05,598 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:05,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-18 18:56:05,850 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1600 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/4f2918ed406e4f428da9d35b9238555e 2023-04-18 18:56:06,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-18 18:56:06,259 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1600 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/acdd7a3a2b4246ee93582be14898ca60 2023-04-18 18:56:06,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/25d3ca8d0f45478db0d8abf15e8cc246 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/25d3ca8d0f45478db0d8abf15e8cc246 2023-04-18 18:56:06,266 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/25d3ca8d0f45478db0d8abf15e8cc246, entries=150, sequenceid=1600, filesize=11.9 K 2023-04-18 18:56:06,266 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/4f2918ed406e4f428da9d35b9238555e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/4f2918ed406e4f428da9d35b9238555e 2023-04-18 18:56:06,269 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/4f2918ed406e4f428da9d35b9238555e, entries=150, sequenceid=1600, filesize=11.9 K 2023-04-18 18:56:06,269 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/acdd7a3a2b4246ee93582be14898ca60 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/acdd7a3a2b4246ee93582be14898ca60 2023-04-18 18:56:06,271 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/acdd7a3a2b4246ee93582be14898ca60, entries=150, sequenceid=1600, filesize=11.9 K 2023-04-18 18:56:06,272 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=26.84 KB/27480 for 814b3c470b4969ca1c95de968f5dec66 in 1246ms, sequenceid=1600, compaction requested=true 2023-04-18 18:56:06,272 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:06,272 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:06,272 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. because compaction request was cancelled 2023-04-18 18:56:06,272 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:06,272 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:06,272 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. after waiting 0 ms 2023-04-18 18:56:06,272 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. because compaction request was cancelled 2023-04-18 18:56:06,272 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:06,272 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:06,272 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 814b3c470b4969ca1c95de968f5dec66 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-18 18:56:06,272 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. because compaction request was cancelled 2023-04-18 18:56:06,272 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:06,272 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=A 2023-04-18 18:56:06,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:06,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=B 2023-04-18 18:56:06,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:06,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 814b3c470b4969ca1c95de968f5dec66, store=C 2023-04-18 18:56:06,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:06,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-18 18:56:06,678 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1607 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/36f02260d7414877b274ab9c03d4ab06 2023-04-18 18:56:07,089 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1607 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/1ae8d6c1bca34fe1940b1035fccf8f8f 2023-04-18 18:56:07,498 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1607 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/6cd0395ce0434e48b10746a709bb1256 2023-04-18 18:56:07,501 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/A/36f02260d7414877b274ab9c03d4ab06 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/36f02260d7414877b274ab9c03d4ab06 2023-04-18 18:56:07,504 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/36f02260d7414877b274ab9c03d4ab06, entries=50, sequenceid=1607, filesize=7.1 K 2023-04-18 18:56:07,504 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/B/1ae8d6c1bca34fe1940b1035fccf8f8f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/1ae8d6c1bca34fe1940b1035fccf8f8f 2023-04-18 18:56:07,508 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/1ae8d6c1bca34fe1940b1035fccf8f8f, entries=50, sequenceid=1607, filesize=7.1 K 2023-04-18 18:56:07,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/.tmp/C/6cd0395ce0434e48b10746a709bb1256 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/6cd0395ce0434e48b10746a709bb1256 2023-04-18 18:56:07,511 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/6cd0395ce0434e48b10746a709bb1256, entries=50, sequenceid=1607, filesize=7.1 K 2023-04-18 18:56:07,512 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 814b3c470b4969ca1c95de968f5dec66 in 1240ms, sequenceid=1607, compaction requested=true 2023-04-18 18:56:07,537 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/461ebbc7e7754e7b906713bef02403b5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/9b4e1b76613f4469b69c4ea7c3b418fd, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b5f3dd6e801c4bf3944f99f66a186b27, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/0ad9257350a1434d8bb67a49a710beec, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/922b3527b0334751904dfc0f5375f805, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/773ece3b323e46b1bc8e3f57e5c52dcc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/d9ea1679842b411bbf360471dd912737, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/3f14261f6f0c4190a3b2ad4be86bc5c0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/04e07920efd9424da834ca6e9f33461e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/875b060be5124ef8be85717a8b33712f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/76b58a9ef82b4290829b3d9a0b82d57e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/7352789bb5e74a2ab530e3bce7f9150d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/adbb3f2b54b0454e9eac5c43e4740df4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/3bad581c53ba47008ef21c65b90be0fc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/9f3c3ffc0aec458da1da795caa7bcaf5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/87b30d6f0b524e469669bf38157f6bf1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/cfab6d83b19842aa8590121b3e7f5fa2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/31050cfe79564201af448e84127f29c3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/722166b5d1a4445aa274d3b45e265dca, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/cf7bc210489c4cfba44bd2c6e25dd0b0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/7c3e1fe04c174fa4a640f586153bbf59, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/26d5eac8fa074e59b6e787df250caa1e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/372b2d4df0434946a45bf31f8fe38ff9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/db53449058b24fd38cfc70e1abe638b2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/a11eb6bfe0904a67a8c6ffe851efc28f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/433f2d2cf69e477d868105bd1903fae1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/7f722600491849fe871964d6b51e905a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/14103793e82c4c89a10f54a6169eb09a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/da817eeb8f2b421ab98879d60e684650, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/2812200484774f3db8280ac60507200b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/0161a97118f74cfa8d01062c97795041, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/2942ac20b7a54451b09e7feb8c40b212, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/07b91115b91a4b3f828538bdea808591, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/980493ba9365453faf67d480c7cd062f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/9da3a4951aca4b78b0af81083c45c2b9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/49e4709b8d3743a7a26dd4afd5fc3fdb, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/e98e46ed57204df6adf7210df10c21c5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/f5f7d0c447b449e7a626f2e324a6c7eb, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/ca0c9dcbc05c4bd29c45239ec6b752a9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/f2d541815fa2438e8e110e5255099db3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/6f3c3426817b4b63a6c7c6426785448a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/8dda2be568f74912bafbf78a1578907b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b8f99df816764e669d6afeedc52cd821, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b7b4a6c2140542ae954645b7f3d76bda, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/a35e47e492024befbb6e7a3b341bc01e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/960bdb439e5747c4be87d262d4ce8964, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/3264b87f8868468e93d03835bce39ac5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b44d3604ff77438dba97a899fa35e92c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/4fa9033f473c489dbabbae8ee669adcf] to archive 2023-04-18 18:56:07,538 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 18:56:07,539 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/461ebbc7e7754e7b906713bef02403b5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/461ebbc7e7754e7b906713bef02403b5 2023-04-18 18:56:07,541 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/9b4e1b76613f4469b69c4ea7c3b418fd to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/9b4e1b76613f4469b69c4ea7c3b418fd 2023-04-18 18:56:07,541 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b5f3dd6e801c4bf3944f99f66a186b27 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b5f3dd6e801c4bf3944f99f66a186b27 2023-04-18 18:56:07,542 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/0ad9257350a1434d8bb67a49a710beec to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/0ad9257350a1434d8bb67a49a710beec 2023-04-18 18:56:07,543 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/922b3527b0334751904dfc0f5375f805 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/922b3527b0334751904dfc0f5375f805 2023-04-18 18:56:07,544 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/773ece3b323e46b1bc8e3f57e5c52dcc to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/773ece3b323e46b1bc8e3f57e5c52dcc 2023-04-18 18:56:07,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-18 18:56:07,545 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/d9ea1679842b411bbf360471dd912737 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/d9ea1679842b411bbf360471dd912737 2023-04-18 18:56:07,546 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/3f14261f6f0c4190a3b2ad4be86bc5c0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/3f14261f6f0c4190a3b2ad4be86bc5c0 2023-04-18 18:56:07,547 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/04e07920efd9424da834ca6e9f33461e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/04e07920efd9424da834ca6e9f33461e 2023-04-18 18:56:07,548 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/875b060be5124ef8be85717a8b33712f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/875b060be5124ef8be85717a8b33712f 2023-04-18 18:56:07,549 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/76b58a9ef82b4290829b3d9a0b82d57e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/76b58a9ef82b4290829b3d9a0b82d57e 2023-04-18 18:56:07,550 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/7352789bb5e74a2ab530e3bce7f9150d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/7352789bb5e74a2ab530e3bce7f9150d 2023-04-18 18:56:07,551 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/adbb3f2b54b0454e9eac5c43e4740df4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/adbb3f2b54b0454e9eac5c43e4740df4 2023-04-18 18:56:07,552 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/3bad581c53ba47008ef21c65b90be0fc to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/3bad581c53ba47008ef21c65b90be0fc 2023-04-18 18:56:07,553 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/9f3c3ffc0aec458da1da795caa7bcaf5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/9f3c3ffc0aec458da1da795caa7bcaf5 2023-04-18 18:56:07,554 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/87b30d6f0b524e469669bf38157f6bf1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/87b30d6f0b524e469669bf38157f6bf1 2023-04-18 18:56:07,555 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/cfab6d83b19842aa8590121b3e7f5fa2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/cfab6d83b19842aa8590121b3e7f5fa2 2023-04-18 18:56:07,556 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/31050cfe79564201af448e84127f29c3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/31050cfe79564201af448e84127f29c3 2023-04-18 18:56:07,557 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/722166b5d1a4445aa274d3b45e265dca to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/722166b5d1a4445aa274d3b45e265dca 2023-04-18 18:56:07,558 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/cf7bc210489c4cfba44bd2c6e25dd0b0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/cf7bc210489c4cfba44bd2c6e25dd0b0 2023-04-18 18:56:07,558 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/7c3e1fe04c174fa4a640f586153bbf59 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/7c3e1fe04c174fa4a640f586153bbf59 2023-04-18 18:56:07,559 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/26d5eac8fa074e59b6e787df250caa1e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/26d5eac8fa074e59b6e787df250caa1e 2023-04-18 18:56:07,560 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/372b2d4df0434946a45bf31f8fe38ff9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/372b2d4df0434946a45bf31f8fe38ff9 2023-04-18 18:56:07,561 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/db53449058b24fd38cfc70e1abe638b2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/db53449058b24fd38cfc70e1abe638b2 2023-04-18 18:56:07,562 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/a11eb6bfe0904a67a8c6ffe851efc28f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/a11eb6bfe0904a67a8c6ffe851efc28f 2023-04-18 18:56:07,563 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/433f2d2cf69e477d868105bd1903fae1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/433f2d2cf69e477d868105bd1903fae1 2023-04-18 18:56:07,564 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/7f722600491849fe871964d6b51e905a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/7f722600491849fe871964d6b51e905a 2023-04-18 18:56:07,565 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/14103793e82c4c89a10f54a6169eb09a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/14103793e82c4c89a10f54a6169eb09a 2023-04-18 18:56:07,566 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/da817eeb8f2b421ab98879d60e684650 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/da817eeb8f2b421ab98879d60e684650 2023-04-18 18:56:07,567 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/2812200484774f3db8280ac60507200b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/2812200484774f3db8280ac60507200b 2023-04-18 18:56:07,568 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/0161a97118f74cfa8d01062c97795041 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/0161a97118f74cfa8d01062c97795041 2023-04-18 18:56:07,569 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/2942ac20b7a54451b09e7feb8c40b212 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/2942ac20b7a54451b09e7feb8c40b212 2023-04-18 18:56:07,570 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/07b91115b91a4b3f828538bdea808591 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/07b91115b91a4b3f828538bdea808591 2023-04-18 18:56:07,570 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/980493ba9365453faf67d480c7cd062f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/980493ba9365453faf67d480c7cd062f 2023-04-18 18:56:07,571 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/9da3a4951aca4b78b0af81083c45c2b9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/9da3a4951aca4b78b0af81083c45c2b9 2023-04-18 18:56:07,572 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/49e4709b8d3743a7a26dd4afd5fc3fdb to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/49e4709b8d3743a7a26dd4afd5fc3fdb 2023-04-18 18:56:07,573 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/e98e46ed57204df6adf7210df10c21c5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/e98e46ed57204df6adf7210df10c21c5 2023-04-18 18:56:07,574 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/f5f7d0c447b449e7a626f2e324a6c7eb to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/f5f7d0c447b449e7a626f2e324a6c7eb 2023-04-18 18:56:07,575 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/ca0c9dcbc05c4bd29c45239ec6b752a9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/ca0c9dcbc05c4bd29c45239ec6b752a9 2023-04-18 18:56:07,575 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/f2d541815fa2438e8e110e5255099db3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/f2d541815fa2438e8e110e5255099db3 2023-04-18 18:56:07,576 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/6f3c3426817b4b63a6c7c6426785448a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/6f3c3426817b4b63a6c7c6426785448a 2023-04-18 18:56:07,577 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/8dda2be568f74912bafbf78a1578907b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/8dda2be568f74912bafbf78a1578907b 2023-04-18 18:56:07,578 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b8f99df816764e669d6afeedc52cd821 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b8f99df816764e669d6afeedc52cd821 2023-04-18 18:56:07,579 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b7b4a6c2140542ae954645b7f3d76bda to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b7b4a6c2140542ae954645b7f3d76bda 2023-04-18 18:56:07,580 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/a35e47e492024befbb6e7a3b341bc01e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/a35e47e492024befbb6e7a3b341bc01e 2023-04-18 18:56:07,581 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/960bdb439e5747c4be87d262d4ce8964 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/960bdb439e5747c4be87d262d4ce8964 2023-04-18 18:56:07,582 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/3264b87f8868468e93d03835bce39ac5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/3264b87f8868468e93d03835bce39ac5 2023-04-18 18:56:07,582 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b44d3604ff77438dba97a899fa35e92c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/b44d3604ff77438dba97a899fa35e92c 2023-04-18 18:56:07,583 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/4fa9033f473c489dbabbae8ee669adcf to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/4fa9033f473c489dbabbae8ee669adcf 2023-04-18 18:56:07,587 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/a01df87a0d524a5183b70a24f8609766, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/f2dcce9f481343deba517775f3ddae28, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8f08bb6524804e65a439c2266cf5e937, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/d332a09c4dcc4611877909baab6f67e1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/fd48d95314c647f383ddc41167f12b61, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/7ad77f28fea74842b30fcd8e8859a51f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/e614f91ce9764f329255e831aad8a116, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8896a7be22ed4ad697da3d30da54a7a4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ec23ef2a32f54dbf9741600d7077fc65, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/76c19dbb6d544fb9a3bfa075f8e71cde, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/6ecfa75ea86d4f6088e4b6d5f2c4cfad, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/4152098fb1a5434cace9f89b49edb280, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/767210ce97bc4277bda2732ebde38889, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/9ad2742c6a774191b0ef8f27a66a62a2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/5ebe0bc8a6614df4b063e36ac3a462a1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/f977335a15324f4e8cde60ed293163f6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/41f59f53c20d49cd8572e6e0869697d9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/c36ebe44abbd4270b41b8b5bfbc8503e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/b253086fd2ce4405ae88d270cdb2f443, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/fadd7dbc8f5042c981c525aff5a758b5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/c5254c10f7044f5c90d93e197a28de81, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8c04539bdbc0461cb31df6fd1bfc2e9c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/98f05764e9834fc091d29eb762698cba, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/e1134d06c3ff4fd39e1701b85d472645, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/2c1a34c06ec24e6297bef1bec0daa3fc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/7c51936041d7463d857a2b1a16c0c374, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/40af8a0829224828ad0b2c21c9d48dcd, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/47288e7124c34fcdbbf1ff9db0ce9814, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/edcd2565a4b147b593f7d0857ac40e63, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/debacd8c5cc54426bb8e2d540284b216, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ffacb2b6c90848838734a44a4106c9b8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/500dfdd33f4e42aaa803bf07b8a36446, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/f97af59c9f524916a35a9985e257ed92, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/37aed5b73362471ab4912aa650df52c1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/a3cf1b44f9bb4738bdd4ed715be29c70, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/04a1d1223fcf4c39945998f9738fe830, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8e80d1fa52e64dc5b6a6fd0e502bda39, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/e3a4f93c371d429dbdf4f32167d936cf, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8dbc73d7ebda4d78b2f46e8fa330224d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/9b0986a8e8194b6a8ed9d9f84770a987, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/d73cf1c67df34644be4d83d34f15834e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/303d3b16cbab45718c2afd33fcadf332, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/18edebbb25e44287bbb647fdfa11a781, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/6ca925517b5c44339371661b9aa96195, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ef3af24d1809403ea6227b30ecb63cd4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/34e2fee7fc0741d2aebe5278fa832b79, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/7af60467bbfe4928b7cd141076467d3a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ed29e20f8fee4d3392f5e7c41f336c02, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/80e45f16b2084660a3f43f256f121c13] to archive 2023-04-18 18:56:07,587 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 18:56:07,589 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/a01df87a0d524a5183b70a24f8609766 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/a01df87a0d524a5183b70a24f8609766 2023-04-18 18:56:07,590 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/f2dcce9f481343deba517775f3ddae28 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/f2dcce9f481343deba517775f3ddae28 2023-04-18 18:56:07,590 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8f08bb6524804e65a439c2266cf5e937 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8f08bb6524804e65a439c2266cf5e937 2023-04-18 18:56:07,591 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/d332a09c4dcc4611877909baab6f67e1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/d332a09c4dcc4611877909baab6f67e1 2023-04-18 18:56:07,592 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/fd48d95314c647f383ddc41167f12b61 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/fd48d95314c647f383ddc41167f12b61 2023-04-18 18:56:07,593 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/7ad77f28fea74842b30fcd8e8859a51f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/7ad77f28fea74842b30fcd8e8859a51f 2023-04-18 18:56:07,594 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/e614f91ce9764f329255e831aad8a116 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/e614f91ce9764f329255e831aad8a116 2023-04-18 18:56:07,595 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8896a7be22ed4ad697da3d30da54a7a4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8896a7be22ed4ad697da3d30da54a7a4 2023-04-18 18:56:07,596 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ec23ef2a32f54dbf9741600d7077fc65 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ec23ef2a32f54dbf9741600d7077fc65 2023-04-18 18:56:07,596 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/76c19dbb6d544fb9a3bfa075f8e71cde to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/76c19dbb6d544fb9a3bfa075f8e71cde 2023-04-18 18:56:07,597 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/6ecfa75ea86d4f6088e4b6d5f2c4cfad to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/6ecfa75ea86d4f6088e4b6d5f2c4cfad 2023-04-18 18:56:07,598 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/4152098fb1a5434cace9f89b49edb280 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/4152098fb1a5434cace9f89b49edb280 2023-04-18 18:56:07,599 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/767210ce97bc4277bda2732ebde38889 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/767210ce97bc4277bda2732ebde38889 2023-04-18 18:56:07,600 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/9ad2742c6a774191b0ef8f27a66a62a2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/9ad2742c6a774191b0ef8f27a66a62a2 2023-04-18 18:56:07,601 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/5ebe0bc8a6614df4b063e36ac3a462a1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/5ebe0bc8a6614df4b063e36ac3a462a1 2023-04-18 18:56:07,601 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/f977335a15324f4e8cde60ed293163f6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/f977335a15324f4e8cde60ed293163f6 2023-04-18 18:56:07,602 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/41f59f53c20d49cd8572e6e0869697d9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/41f59f53c20d49cd8572e6e0869697d9 2023-04-18 18:56:07,603 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/c36ebe44abbd4270b41b8b5bfbc8503e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/c36ebe44abbd4270b41b8b5bfbc8503e 2023-04-18 18:56:07,604 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/b253086fd2ce4405ae88d270cdb2f443 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/b253086fd2ce4405ae88d270cdb2f443 2023-04-18 18:56:07,605 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/fadd7dbc8f5042c981c525aff5a758b5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/fadd7dbc8f5042c981c525aff5a758b5 2023-04-18 18:56:07,606 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/c5254c10f7044f5c90d93e197a28de81 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/c5254c10f7044f5c90d93e197a28de81 2023-04-18 18:56:07,606 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8c04539bdbc0461cb31df6fd1bfc2e9c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8c04539bdbc0461cb31df6fd1bfc2e9c 2023-04-18 18:56:07,607 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/98f05764e9834fc091d29eb762698cba to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/98f05764e9834fc091d29eb762698cba 2023-04-18 18:56:07,608 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/e1134d06c3ff4fd39e1701b85d472645 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/e1134d06c3ff4fd39e1701b85d472645 2023-04-18 18:56:07,609 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/2c1a34c06ec24e6297bef1bec0daa3fc to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/2c1a34c06ec24e6297bef1bec0daa3fc 2023-04-18 18:56:07,610 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/7c51936041d7463d857a2b1a16c0c374 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/7c51936041d7463d857a2b1a16c0c374 2023-04-18 18:56:07,610 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/40af8a0829224828ad0b2c21c9d48dcd to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/40af8a0829224828ad0b2c21c9d48dcd 2023-04-18 18:56:07,611 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/47288e7124c34fcdbbf1ff9db0ce9814 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/47288e7124c34fcdbbf1ff9db0ce9814 2023-04-18 18:56:07,612 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/edcd2565a4b147b593f7d0857ac40e63 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/edcd2565a4b147b593f7d0857ac40e63 2023-04-18 18:56:07,613 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/debacd8c5cc54426bb8e2d540284b216 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/debacd8c5cc54426bb8e2d540284b216 2023-04-18 18:56:07,614 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ffacb2b6c90848838734a44a4106c9b8 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ffacb2b6c90848838734a44a4106c9b8 2023-04-18 18:56:07,615 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/500dfdd33f4e42aaa803bf07b8a36446 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/500dfdd33f4e42aaa803bf07b8a36446 2023-04-18 18:56:07,615 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/f97af59c9f524916a35a9985e257ed92 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/f97af59c9f524916a35a9985e257ed92 2023-04-18 18:56:07,616 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/37aed5b73362471ab4912aa650df52c1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/37aed5b73362471ab4912aa650df52c1 2023-04-18 18:56:07,617 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/a3cf1b44f9bb4738bdd4ed715be29c70 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/a3cf1b44f9bb4738bdd4ed715be29c70 2023-04-18 18:56:07,618 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/04a1d1223fcf4c39945998f9738fe830 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/04a1d1223fcf4c39945998f9738fe830 2023-04-18 18:56:07,619 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8e80d1fa52e64dc5b6a6fd0e502bda39 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8e80d1fa52e64dc5b6a6fd0e502bda39 2023-04-18 18:56:07,620 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/e3a4f93c371d429dbdf4f32167d936cf to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/e3a4f93c371d429dbdf4f32167d936cf 2023-04-18 18:56:07,620 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8dbc73d7ebda4d78b2f46e8fa330224d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8dbc73d7ebda4d78b2f46e8fa330224d 2023-04-18 18:56:07,621 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/9b0986a8e8194b6a8ed9d9f84770a987 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/9b0986a8e8194b6a8ed9d9f84770a987 2023-04-18 18:56:07,622 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/d73cf1c67df34644be4d83d34f15834e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/d73cf1c67df34644be4d83d34f15834e 2023-04-18 18:56:07,623 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/303d3b16cbab45718c2afd33fcadf332 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/303d3b16cbab45718c2afd33fcadf332 2023-04-18 18:56:07,624 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/18edebbb25e44287bbb647fdfa11a781 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/18edebbb25e44287bbb647fdfa11a781 2023-04-18 18:56:07,624 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/6ca925517b5c44339371661b9aa96195 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/6ca925517b5c44339371661b9aa96195 2023-04-18 18:56:07,625 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ef3af24d1809403ea6227b30ecb63cd4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ef3af24d1809403ea6227b30ecb63cd4 2023-04-18 18:56:07,627 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/34e2fee7fc0741d2aebe5278fa832b79 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/34e2fee7fc0741d2aebe5278fa832b79 2023-04-18 18:56:07,627 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/7af60467bbfe4928b7cd141076467d3a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/7af60467bbfe4928b7cd141076467d3a 2023-04-18 18:56:07,628 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ed29e20f8fee4d3392f5e7c41f336c02 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ed29e20f8fee4d3392f5e7c41f336c02 2023-04-18 18:56:07,629 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/80e45f16b2084660a3f43f256f121c13 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/80e45f16b2084660a3f43f256f121c13 2023-04-18 18:56:07,633 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/3542937df8644932baf310cf457aab84, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/7cbb0bd13be54e298ce09994fcd1d2d7, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/69ab700726de43639d4c881f7866d5b6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/182148ae60634e019470f59bbab0837d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/b481be1090bd446294eeac809fe27d3d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f2cc4126afd047de9323655442999a2a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/620785e7930e4e16a578dc4351a8f001, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/1abcb70242c448ea83daf3fc2825fca8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/e9d2b44108a6464ebca004909ee94d17, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/aa43993c73e34bf98cfc7508810b232c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/6c8f018ef9104262a0eab002b7725bf8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/1b0b03d89e294496a7404c234be15add, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/62ae2d8298f642c981b989126808830d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f8a13fa737214ae1b84f35213bf333e3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f90a088424964fb692056d4be83c242c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/47238a0f2a554909bcd2aa62bdb4d153, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f67cf587c48b493c826cb6548975ab92, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/b9179739a8a84945b53eb6e5b8aa3092, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/3ee3767e85ed4e319c4d97806bfddf86, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/adc459e664974b4db62421bef63671b9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/fdf754176e1842be887864ec041b8566, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/db9638f1c4ad42c796f474ded5df240b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f7b60a88bda5418d8e9570881a0835d5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/bb0354d8e09c48889501d4442d7be77f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/6bae45fc63cf405594aafaf79499ffa3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/3bcf86f2e7294312bcc39671d0d0fdb5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/579f859a4748401db218b7567108b53a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/26d1fb3329b14ce688f6bbb3f1bdc8cd, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/176e087ef2094ec7adc0024f26039d4e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/24bdfcbfaecd45868696a8d2c1690f0c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/49628cafe1cb4e608dd4920a133eb8d9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/ade1ebe765dc432685d5e84c3bad704d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/e2dc6cf456834b07a81aee271c88e8f1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/7f1828bcafa141959f50fa11a790d954, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/af648e86e37641009c4770bfdff8fc1d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/e8d5d33b38bf41ee8697653f5e73612d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/58b159a091924a68b4173c292fe9ed14, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/55e300a57e4f4c4f9cc0a9a41403c776, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/9aeccab4eb7a4dddbf540a61ade98206, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/d5c94635be2d417c849122638d599ad9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/5dc07cad03974bd79c494cc237b1a985, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/85eada4a65b44346aca6266fc56894e5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f621c8b4b1cc463988b81aa920680df0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/a2953d0b4315499285084494db172d9d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/d1dfcc4110204369b0eecca204cdf3e2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/2b777b45249c4d09942ffa48ebdd62f1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/adc5017262ea40788132865fcebad743, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/bb063c7340f04a9db0d82c4735ba81be, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/2f1a8848c2254c7581c075c9a9f3fab5] to archive 2023-04-18 18:56:07,633 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 18:56:07,634 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/3542937df8644932baf310cf457aab84 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/3542937df8644932baf310cf457aab84 2023-04-18 18:56:07,635 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/7cbb0bd13be54e298ce09994fcd1d2d7 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/7cbb0bd13be54e298ce09994fcd1d2d7 2023-04-18 18:56:07,636 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/69ab700726de43639d4c881f7866d5b6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/69ab700726de43639d4c881f7866d5b6 2023-04-18 18:56:07,637 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/182148ae60634e019470f59bbab0837d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/182148ae60634e019470f59bbab0837d 2023-04-18 18:56:07,637 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/b481be1090bd446294eeac809fe27d3d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/b481be1090bd446294eeac809fe27d3d 2023-04-18 18:56:07,638 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f2cc4126afd047de9323655442999a2a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f2cc4126afd047de9323655442999a2a 2023-04-18 18:56:07,639 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/620785e7930e4e16a578dc4351a8f001 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/620785e7930e4e16a578dc4351a8f001 2023-04-18 18:56:07,639 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/1abcb70242c448ea83daf3fc2825fca8 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/1abcb70242c448ea83daf3fc2825fca8 2023-04-18 18:56:07,640 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/e9d2b44108a6464ebca004909ee94d17 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/e9d2b44108a6464ebca004909ee94d17 2023-04-18 18:56:07,641 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/aa43993c73e34bf98cfc7508810b232c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/aa43993c73e34bf98cfc7508810b232c 2023-04-18 18:56:07,642 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/6c8f018ef9104262a0eab002b7725bf8 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/6c8f018ef9104262a0eab002b7725bf8 2023-04-18 18:56:07,642 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/1b0b03d89e294496a7404c234be15add to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/1b0b03d89e294496a7404c234be15add 2023-04-18 18:56:07,643 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/62ae2d8298f642c981b989126808830d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/62ae2d8298f642c981b989126808830d 2023-04-18 18:56:07,644 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f8a13fa737214ae1b84f35213bf333e3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f8a13fa737214ae1b84f35213bf333e3 2023-04-18 18:56:07,644 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f90a088424964fb692056d4be83c242c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f90a088424964fb692056d4be83c242c 2023-04-18 18:56:07,645 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/47238a0f2a554909bcd2aa62bdb4d153 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/47238a0f2a554909bcd2aa62bdb4d153 2023-04-18 18:56:07,646 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f67cf587c48b493c826cb6548975ab92 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f67cf587c48b493c826cb6548975ab92 2023-04-18 18:56:07,647 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/b9179739a8a84945b53eb6e5b8aa3092 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/b9179739a8a84945b53eb6e5b8aa3092 2023-04-18 18:56:07,647 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/3ee3767e85ed4e319c4d97806bfddf86 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/3ee3767e85ed4e319c4d97806bfddf86 2023-04-18 18:56:07,648 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/adc459e664974b4db62421bef63671b9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/adc459e664974b4db62421bef63671b9 2023-04-18 18:56:07,649 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/fdf754176e1842be887864ec041b8566 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/fdf754176e1842be887864ec041b8566 2023-04-18 18:56:07,650 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/db9638f1c4ad42c796f474ded5df240b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/db9638f1c4ad42c796f474ded5df240b 2023-04-18 18:56:07,650 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f7b60a88bda5418d8e9570881a0835d5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f7b60a88bda5418d8e9570881a0835d5 2023-04-18 18:56:07,651 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/bb0354d8e09c48889501d4442d7be77f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/bb0354d8e09c48889501d4442d7be77f 2023-04-18 18:56:07,652 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/6bae45fc63cf405594aafaf79499ffa3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/6bae45fc63cf405594aafaf79499ffa3 2023-04-18 18:56:07,653 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/3bcf86f2e7294312bcc39671d0d0fdb5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/3bcf86f2e7294312bcc39671d0d0fdb5 2023-04-18 18:56:07,653 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/579f859a4748401db218b7567108b53a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/579f859a4748401db218b7567108b53a 2023-04-18 18:56:07,654 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/26d1fb3329b14ce688f6bbb3f1bdc8cd to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/26d1fb3329b14ce688f6bbb3f1bdc8cd 2023-04-18 18:56:07,655 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/176e087ef2094ec7adc0024f26039d4e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/176e087ef2094ec7adc0024f26039d4e 2023-04-18 18:56:07,655 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/24bdfcbfaecd45868696a8d2c1690f0c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/24bdfcbfaecd45868696a8d2c1690f0c 2023-04-18 18:56:07,656 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/49628cafe1cb4e608dd4920a133eb8d9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/49628cafe1cb4e608dd4920a133eb8d9 2023-04-18 18:56:07,657 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/ade1ebe765dc432685d5e84c3bad704d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/ade1ebe765dc432685d5e84c3bad704d 2023-04-18 18:56:07,658 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/e2dc6cf456834b07a81aee271c88e8f1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/e2dc6cf456834b07a81aee271c88e8f1 2023-04-18 18:56:07,658 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/7f1828bcafa141959f50fa11a790d954 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/7f1828bcafa141959f50fa11a790d954 2023-04-18 18:56:07,659 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/af648e86e37641009c4770bfdff8fc1d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/af648e86e37641009c4770bfdff8fc1d 2023-04-18 18:56:07,660 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/e8d5d33b38bf41ee8697653f5e73612d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/e8d5d33b38bf41ee8697653f5e73612d 2023-04-18 18:56:07,661 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/58b159a091924a68b4173c292fe9ed14 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/58b159a091924a68b4173c292fe9ed14 2023-04-18 18:56:07,661 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/55e300a57e4f4c4f9cc0a9a41403c776 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/55e300a57e4f4c4f9cc0a9a41403c776 2023-04-18 18:56:07,662 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/9aeccab4eb7a4dddbf540a61ade98206 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/9aeccab4eb7a4dddbf540a61ade98206 2023-04-18 18:56:07,663 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/d5c94635be2d417c849122638d599ad9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/d5c94635be2d417c849122638d599ad9 2023-04-18 18:56:07,663 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/5dc07cad03974bd79c494cc237b1a985 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/5dc07cad03974bd79c494cc237b1a985 2023-04-18 18:56:07,664 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/85eada4a65b44346aca6266fc56894e5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/85eada4a65b44346aca6266fc56894e5 2023-04-18 18:56:07,665 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f621c8b4b1cc463988b81aa920680df0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/f621c8b4b1cc463988b81aa920680df0 2023-04-18 18:56:07,666 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/a2953d0b4315499285084494db172d9d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/a2953d0b4315499285084494db172d9d 2023-04-18 18:56:07,667 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/d1dfcc4110204369b0eecca204cdf3e2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/d1dfcc4110204369b0eecca204cdf3e2 2023-04-18 18:56:07,667 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/2b777b45249c4d09942ffa48ebdd62f1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/2b777b45249c4d09942ffa48ebdd62f1 2023-04-18 18:56:07,668 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/adc5017262ea40788132865fcebad743 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/adc5017262ea40788132865fcebad743 2023-04-18 18:56:07,669 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/bb063c7340f04a9db0d82c4735ba81be to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/bb063c7340f04a9db0d82c4735ba81be 2023-04-18 18:56:07,670 DEBUG [StoreCloser-TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/2f1a8848c2254c7581c075c9a9f3fab5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/2f1a8848c2254c7581c075c9a9f3fab5 2023-04-18 18:56:07,673 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/recovered.edits/1610.seqid, newMaxSeqId=1610, maxSeqId=1 2023-04-18 18:56:07,675 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66. 2023-04-18 18:56:07,675 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 814b3c470b4969ca1c95de968f5dec66: 2023-04-18 18:56:07,676 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:56:07,676 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=814b3c470b4969ca1c95de968f5dec66, regionState=CLOSED 2023-04-18 18:56:07,677 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844167676"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681844167676"}]},"ts":"1681844167676"} 2023-04-18 18:56:07,678 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-18 18:56:07,678 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 814b3c470b4969ca1c95de968f5dec66, server=jenkins-hbase4.apache.org,34053,1681844018952 in 2.2310 sec 2023-04-18 18:56:07,679 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-18 18:56:07,679 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=814b3c470b4969ca1c95de968f5dec66, UNASSIGN in 2.2350 sec 2023-04-18 18:56:07,680 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844167680"}]},"ts":"1681844167680"} 2023-04-18 18:56:07,680 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-18 18:56:07,683 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-18 18:56:07,684 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.2470 sec 2023-04-18 18:56:09,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-18 18:56:09,546 INFO [Listener at localhost/36951] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-18 18:56:09,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-18 18:56:09,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:56:09,548 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:56:09,549 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:56:09,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-18 18:56:09,552 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:56:09,554 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A, FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B, FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C, FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/recovered.edits] 2023-04-18 18:56:09,556 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/25d3ca8d0f45478db0d8abf15e8cc246 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/25d3ca8d0f45478db0d8abf15e8cc246 2023-04-18 18:56:09,557 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/36f02260d7414877b274ab9c03d4ab06 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/36f02260d7414877b274ab9c03d4ab06 2023-04-18 18:56:09,558 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/81a9fb08d90942738a05d7fff7db429d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/81a9fb08d90942738a05d7fff7db429d 2023-04-18 18:56:09,559 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/e7519d97d8384ed9b0d97eecc74399f7 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/A/e7519d97d8384ed9b0d97eecc74399f7 2023-04-18 18:56:09,561 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/1ae8d6c1bca34fe1940b1035fccf8f8f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/1ae8d6c1bca34fe1940b1035fccf8f8f 2023-04-18 18:56:09,562 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/4f2918ed406e4f428da9d35b9238555e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/4f2918ed406e4f428da9d35b9238555e 2023-04-18 18:56:09,563 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8cc90d941fb1474eba0d091c55944271 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/8cc90d941fb1474eba0d091c55944271 2023-04-18 18:56:09,564 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ccb9d88614be4eaba914c3ebff62afa4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/B/ccb9d88614be4eaba914c3ebff62afa4 2023-04-18 18:56:09,566 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/31dd6e37b6b447698eb7e70e7561fc60 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/31dd6e37b6b447698eb7e70e7561fc60 2023-04-18 18:56:09,567 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/6cd0395ce0434e48b10746a709bb1256 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/6cd0395ce0434e48b10746a709bb1256 2023-04-18 18:56:09,568 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/79e358889c064432833364c3e2fb3463 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/79e358889c064432833364c3e2fb3463 2023-04-18 18:56:09,569 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/acdd7a3a2b4246ee93582be14898ca60 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/C/acdd7a3a2b4246ee93582be14898ca60 2023-04-18 18:56:09,571 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/recovered.edits/1610.seqid to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66/recovered.edits/1610.seqid 2023-04-18 18:56:09,571 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/814b3c470b4969ca1c95de968f5dec66 2023-04-18 18:56:09,572 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 18:56:09,573 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:56:09,574 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-18 18:56:09,576 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-18 18:56:09,577 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:56:09,577 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-18 18:56:09,577 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681844169577"}]},"ts":"9223372036854775807"} 2023-04-18 18:56:09,579 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-18 18:56:09,579 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 814b3c470b4969ca1c95de968f5dec66, NAME => 'TestAcidGuarantees,,1681844142183.814b3c470b4969ca1c95de968f5dec66.', STARTKEY => '', ENDKEY => ''}] 2023-04-18 18:56:09,579 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-18 18:56:09,579 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681844169579"}]},"ts":"9223372036854775807"} 2023-04-18 18:56:09,580 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-18 18:56:09,582 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:56:09,583 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 36 msec 2023-04-18 18:56:09,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-18 18:56:09,650 INFO [Listener at localhost/36951] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-18 18:56:09,663 INFO [Listener at localhost/36951] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=228 (was 229), OpenFileDescriptor=610 (was 517) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=386 (was 320) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 172), AvailableMemoryMB=6909 (was 6427) - AvailableMemoryMB LEAK? - 2023-04-18 18:56:09,678 INFO [Listener at localhost/36951] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228, OpenFileDescriptor=610, MaxFileDescriptor=60000, SystemLoadAverage=386, ProcessCount=170, AvailableMemoryMB=6909 2023-04-18 18:56:09,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-18 18:56:09,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-18 18:56:09,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-18 18:56:09,683 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-18 18:56:09,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-18 18:56:09,683 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-18 18:56:09,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-18 18:56:09,685 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:09,685 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd empty. 2023-04-18 18:56:09,685 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:09,685 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 18:56:09,691 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-18 18:56:09,693 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => c48561c851d33e29da879cafb393a8fd, NAME => 'TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp 2023-04-18 18:56:09,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-18 18:56:09,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-18 18:56:10,097 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 18:56:10,097 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing c48561c851d33e29da879cafb393a8fd, disabling compactions & flushes 2023-04-18 18:56:10,097 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:10,097 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:10,097 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. after waiting 0 ms 2023-04-18 18:56:10,097 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:10,097 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:10,098 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:10,099 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-18 18:56:10,100 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844170100"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681844170100"}]},"ts":"1681844170100"} 2023-04-18 18:56:10,101 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-18 18:56:10,101 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-18 18:56:10,102 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844170102"}]},"ts":"1681844170102"} 2023-04-18 18:56:10,102 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-18 18:56:10,105 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c48561c851d33e29da879cafb393a8fd, ASSIGN}] 2023-04-18 18:56:10,106 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c48561c851d33e29da879cafb393a8fd, ASSIGN 2023-04-18 18:56:10,107 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c48561c851d33e29da879cafb393a8fd, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34053,1681844018952; forceNewPlan=false, retain=false 2023-04-18 18:56:10,258 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=c48561c851d33e29da879cafb393a8fd, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:10,258 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844170258"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681844170258"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681844170258"}]},"ts":"1681844170258"} 2023-04-18 18:56:10,259 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952}] 2023-04-18 18:56:10,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-18 18:56:10,414 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:10,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c48561c851d33e29da879cafb393a8fd, NAME => 'TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.', STARTKEY => '', ENDKEY => ''} 2023-04-18 18:56:10,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:10,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 18:56:10,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:10,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:10,415 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:10,416 DEBUG [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A 2023-04-18 18:56:10,416 DEBUG [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A 2023-04-18 18:56:10,417 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:56:10,417 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c48561c851d33e29da879cafb393a8fd columnFamilyName A 2023-04-18 18:56:10,417 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] regionserver.HStore(310): Store=c48561c851d33e29da879cafb393a8fd/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:56:10,417 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:10,418 DEBUG [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B 2023-04-18 18:56:10,418 DEBUG [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B 2023-04-18 18:56:10,419 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:56:10,419 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c48561c851d33e29da879cafb393a8fd columnFamilyName B 2023-04-18 18:56:10,420 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] regionserver.HStore(310): Store=c48561c851d33e29da879cafb393a8fd/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:56:10,420 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:10,421 DEBUG [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C 2023-04-18 18:56:10,421 DEBUG [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C 2023-04-18 18:56:10,421 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:56:10,421 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c48561c851d33e29da879cafb393a8fd columnFamilyName C 2023-04-18 18:56:10,422 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] regionserver.HStore(310): Store=c48561c851d33e29da879cafb393a8fd/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:56:10,422 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:10,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:10,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:10,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 18:56:10,425 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:10,427 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-18 18:56:10,427 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c48561c851d33e29da879cafb393a8fd; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73916379, jitterRate=0.10143987834453583}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 18:56:10,427 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:10,428 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., pid=56, masterSystemTime=1681844170411 2023-04-18 18:56:10,429 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:10,429 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:10,429 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=c48561c851d33e29da879cafb393a8fd, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:10,429 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844170429"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681844170429"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681844170429"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681844170429"}]},"ts":"1681844170429"} 2023-04-18 18:56:10,431 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-18 18:56:10,431 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 in 171 msec 2023-04-18 18:56:10,432 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-18 18:56:10,432 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c48561c851d33e29da879cafb393a8fd, ASSIGN in 326 msec 2023-04-18 18:56:10,432 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-18 18:56:10,432 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844170432"}]},"ts":"1681844170432"} 2023-04-18 18:56:10,433 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-18 18:56:10,435 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-18 18:56:10,436 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 755 msec 2023-04-18 18:56:10,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-18 18:56:10,788 INFO [Listener at localhost/36951] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-18 18:56:10,788 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x0c0766f0 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:56:10,794 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b13e8aa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:56:10,795 DEBUG [Listener at localhost/36951] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:56:10,796 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43852, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:56:10,797 DEBUG [Listener at localhost/36951] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-18 18:56:10,799 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52780, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-18 18:56:10,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-18 18:56:10,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-18 18:56:10,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-18 18:56:11,214 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-18 18:56:11,215 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-18 18:56:11,215 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-18 18:56:11,217 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-18 18:56:11,219 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c48561c851d33e29da879cafb393a8fd, REOPEN/MOVE}] 2023-04-18 18:56:11,219 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c48561c851d33e29da879cafb393a8fd, REOPEN/MOVE 2023-04-18 18:56:11,220 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=c48561c851d33e29da879cafb393a8fd, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,220 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844171220"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681844171220"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681844171220"}]},"ts":"1681844171220"} 2023-04-18 18:56:11,221 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952}] 2023-04-18 18:56:11,373 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:11,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c48561c851d33e29da879cafb393a8fd, disabling compactions & flushes 2023-04-18 18:56:11,374 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:11,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:11,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. after waiting 0 ms 2023-04-18 18:56:11,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:11,378 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-18 18:56:11,379 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:11,379 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:11,379 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: c48561c851d33e29da879cafb393a8fd to self. 2023-04-18 18:56:11,380 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:11,380 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=c48561c851d33e29da879cafb393a8fd, regionState=CLOSED 2023-04-18 18:56:11,380 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844171380"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681844171380"}]},"ts":"1681844171380"} 2023-04-18 18:56:11,382 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-18 18:56:11,382 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 in 160 msec 2023-04-18 18:56:11,382 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c48561c851d33e29da879cafb393a8fd, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,34053,1681844018952; forceNewPlan=false, retain=true 2023-04-18 18:56:11,533 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=c48561c851d33e29da879cafb393a8fd, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,533 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844171533"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681844171533"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681844171533"}]},"ts":"1681844171533"} 2023-04-18 18:56:11,534 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952}] 2023-04-18 18:56:11,689 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:11,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c48561c851d33e29da879cafb393a8fd, NAME => 'TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.', STARTKEY => '', ENDKEY => ''} 2023-04-18 18:56:11,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:11,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 18:56:11,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:11,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:11,691 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:11,692 DEBUG [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A 2023-04-18 18:56:11,692 DEBUG [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A 2023-04-18 18:56:11,692 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:56:11,692 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c48561c851d33e29da879cafb393a8fd columnFamilyName A 2023-04-18 18:56:11,694 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] regionserver.HStore(310): Store=c48561c851d33e29da879cafb393a8fd/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:56:11,694 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:11,695 DEBUG [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B 2023-04-18 18:56:11,695 DEBUG [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B 2023-04-18 18:56:11,695 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:56:11,695 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c48561c851d33e29da879cafb393a8fd columnFamilyName B 2023-04-18 18:56:11,695 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] regionserver.HStore(310): Store=c48561c851d33e29da879cafb393a8fd/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:56:11,695 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:11,696 DEBUG [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C 2023-04-18 18:56:11,696 DEBUG [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C 2023-04-18 18:56:11,696 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 18:56:11,696 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c48561c851d33e29da879cafb393a8fd columnFamilyName C 2023-04-18 18:56:11,697 INFO [StoreOpener-c48561c851d33e29da879cafb393a8fd-1] regionserver.HStore(310): Store=c48561c851d33e29da879cafb393a8fd/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 18:56:11,697 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:11,698 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:11,699 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:11,700 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 18:56:11,701 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:11,702 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c48561c851d33e29da879cafb393a8fd; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66900917, jitterRate=-0.0030986517667770386}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 18:56:11,702 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:11,702 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., pid=61, masterSystemTime=1681844171685 2023-04-18 18:56:11,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:11,703 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:11,704 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=c48561c851d33e29da879cafb393a8fd, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,704 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844171704"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681844171704"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681844171704"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681844171704"}]},"ts":"1681844171704"} 2023-04-18 18:56:11,705 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-18 18:56:11,705 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 in 171 msec 2023-04-18 18:56:11,706 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-18 18:56:11,706 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c48561c851d33e29da879cafb393a8fd, REOPEN/MOVE in 486 msec 2023-04-18 18:56:11,708 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-18 18:56:11,708 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 490 msec 2023-04-18 18:56:11,709 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 907 msec 2023-04-18 18:56:11,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-18 18:56:11,711 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x6257294f to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:56:11,716 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55d0f313, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:56:11,716 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x52104714 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:56:11,725 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9d61148, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:56:11,726 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x4a320817 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:56:11,730 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a4acea2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:56:11,730 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x19c1e3ee to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:56:11,736 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43693271, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:56:11,736 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x526f2221 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:56:11,740 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44277511, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:56:11,741 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x3c1a68cd to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:56:11,747 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43ea1da2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:56:11,747 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x5a709b21 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:56:11,753 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@35c3cca2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:56:11,753 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x156e4ba6 to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:56:11,760 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b1973b9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:56:11,760 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x785e366b to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:56:11,765 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f0a71bd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:56:11,765 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(139): Connect 0x5fdfef3b to 127.0.0.1:65091 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 18:56:11,769 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56bb77c5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 18:56:11,780 DEBUG [hconnection-0x410853bd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:56:11,781 DEBUG [hconnection-0x5f5ac872-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:56:11,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:11,783 DEBUG [hconnection-0x5d9c7890-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:56:11,783 DEBUG [hconnection-0x3551b979-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:56:11,783 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43862, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:56:11,784 DEBUG [hconnection-0x7220ec55-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:56:11,784 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43876, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:56:11,785 DEBUG [hconnection-0x54ed4dae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:56:11,785 DEBUG [hconnection-0x52f73bb2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:56:11,787 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43904, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:56:11,793 DEBUG [hconnection-0x73216b4a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:56:11,793 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43886, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:56:11,793 DEBUG [hconnection-0x173246c2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:56:11,794 DEBUG [hconnection-0x17a79ce6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 18:56:11,794 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43878, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:56:11,798 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43888, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:56:11,798 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43916, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:56:11,798 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43936, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:56:11,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:11,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:11,803 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:11,803 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:11,803 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:11,803 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:11,803 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43922, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:56:11,803 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:11,803 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:11,809 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43952, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 18:56:11,810 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:11,810 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:11,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:11,810 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,811 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:11,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:11,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:11,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:11,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:11,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:11,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:11,814 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:11,814 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:11,814 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:11,820 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:11,821 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:11,821 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:11,821 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:11,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:11,821 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:11,821 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:11,821 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-18 18:56:11,821 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:11,821 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:11,821 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,822 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:11,822 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:11,822 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:11,822 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:11,822 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:11,822 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:11,822 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:11,822 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:11,823 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:11,823 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,824 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:11,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:11,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:11,824 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:11,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:11,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:11,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:11,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:11,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:11,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@349e10aa[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:11,826 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:11,826 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:11,827 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:11,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:11,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:11,827 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:11,827 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:11,827 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:11,828 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,828 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:11,831 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,831 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:11,831 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:11,832 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:11,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:11,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:11,835 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:11,835 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:11,835 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:11,835 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:11,835 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:11,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:11,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844231834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:11,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844231834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:11,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844231834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:11,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844231835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:11,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:11,838 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:11,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:11,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:11,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:11,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844231836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,839 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,840 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:11,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:11,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,843 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:11,843 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:11,843 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:11,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:11,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:11,843 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,843 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:11,843 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:11,843 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,843 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:11,843 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:11,843 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:11,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:11,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:11,844 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:11,844 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:11,857 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418e9feef5f70b340a5bdbcc3e6cbbf655a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e9feef5f70b340a5bdbcc3e6cbbf655a 2023-04-18 18:56:11,865 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/9e367509f9094d9d989c77c04f1f4829 2023-04-18 18:56:11,881 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/0eff99bcdbd746f687208d0a8061f09e 2023-04-18 18:56:11,897 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/9c17dc07c85c48bf82ec6b1816214534 2023-04-18 18:56:11,902 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/9e367509f9094d9d989c77c04f1f4829 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/9e367509f9094d9d989c77c04f1f4829 2023-04-18 18:56:11,905 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/9e367509f9094d9d989c77c04f1f4829, entries=200, sequenceid=17, filesize=31.9 K 2023-04-18 18:56:11,906 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/0eff99bcdbd746f687208d0a8061f09e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/0eff99bcdbd746f687208d0a8061f09e 2023-04-18 18:56:11,909 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/0eff99bcdbd746f687208d0a8061f09e, entries=150, sequenceid=17, filesize=11.6 K 2023-04-18 18:56:11,910 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/9c17dc07c85c48bf82ec6b1816214534 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9c17dc07c85c48bf82ec6b1816214534 2023-04-18 18:56:11,913 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9c17dc07c85c48bf82ec6b1816214534, entries=150, sequenceid=17, filesize=11.6 K 2023-04-18 18:56:11,914 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for c48561c851d33e29da879cafb393a8fd in 93ms, sequenceid=17, compaction requested=false 2023-04-18 18:56:11,914 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:11,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:11,940 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 18:56:11,941 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:11,941 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:11,941 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:11,941 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:11,941 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:11,941 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:11,944 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:11,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:11,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:11,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:11,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:11,948 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:11,948 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:11,948 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:11,948 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:11,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:11,948 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:11,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:11,948 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:11,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844231944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844231944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:11,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:11,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844231944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844231945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:11,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844231947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,950 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:11,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:11,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:11,950 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:11,950 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,950 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:11,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:11,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:11,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:11,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:11,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:11,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:11,952 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:11,952 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:11,952 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:11,952 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:11,952 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:11,952 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:11,952 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:11,952 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:11,952 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:11,952 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:11,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:11,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:11,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:11,954 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,954 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:11,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:11,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:11,954 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:11,955 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:11,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:11,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:11,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:11,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:11,956 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304185b725b46278c40c9b4daf8d9935b5f24 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304185b725b46278c40c9b4daf8d9935b5f24 2023-04-18 18:56:11,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:11,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a0fcb68[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:11,956 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:11,956 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:11,959 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:11,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:11,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:11,959 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,959 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:11,959 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:11,959 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:11,959 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:11,961 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=40, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/f9153095797c430fab62ca852f6ad1f7 2023-04-18 18:56:11,961 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:11,962 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:11,962 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:11,962 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:11,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:11,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:11,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:11,965 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:11,965 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:11,965 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:11,965 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:11,965 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:11,967 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:11,967 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:11,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:11,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:11,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:11,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:11,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:11,968 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:11,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:11,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:11,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:11,969 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:11,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:11,970 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:11,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:11,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,972 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=40 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/59d6a9882f194717872cc7827af5e18b 2023-04-18 18:56:11,973 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,973 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:11,973 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:11,973 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:11,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:11,974 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:11,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:11,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:11,974 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:11,974 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:11,974 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:11,974 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:11,974 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:11,976 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:11,976 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:11,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:11,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:11,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:11,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:12,004 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=40 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/c592bb64f5fc420aa49be513f17e4ef5 2023-04-18 18:56:12,013 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/f9153095797c430fab62ca852f6ad1f7 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/f9153095797c430fab62ca852f6ad1f7 2023-04-18 18:56:12,017 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/f9153095797c430fab62ca852f6ad1f7, entries=200, sequenceid=40, filesize=31.9 K 2023-04-18 18:56:12,018 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/59d6a9882f194717872cc7827af5e18b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/59d6a9882f194717872cc7827af5e18b 2023-04-18 18:56:12,022 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/59d6a9882f194717872cc7827af5e18b, entries=150, sequenceid=40, filesize=11.6 K 2023-04-18 18:56:12,022 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/c592bb64f5fc420aa49be513f17e4ef5 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/c592bb64f5fc420aa49be513f17e4ef5 2023-04-18 18:56:12,029 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/c592bb64f5fc420aa49be513f17e4ef5, entries=150, sequenceid=40, filesize=11.6 K 2023-04-18 18:56:12,030 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~399.84 KB/409440, currentSize=40.25 KB/41220 for c48561c851d33e29da879cafb393a8fd in 90ms, sequenceid=40, compaction requested=false 2023-04-18 18:56:12,030 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:12,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:12,053 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 18:56:12,054 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:12,054 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:12,054 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:12,054 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:12,054 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:12,054 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:12,074 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:12,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681844172066, max timestamp=1681844172073Num uniques 859; 2023-04-18 18:56:12,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 18:56:12,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:12,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:12,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:12,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:12,077 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:12,077 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:12,077 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:12,077 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:12,078 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,078 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:12,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844232075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:12,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844232075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:12,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844232076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,080 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,080 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:12,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:12,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:12,080 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,080 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:12,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:12,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844232078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:12,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844232079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:12,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:12,088 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:12,088 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:12,088 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:12,089 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:12,089 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:12,089 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:12,089 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:12,090 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:12,090 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:12,090 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:12,090 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:12,090 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,090 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:12,090 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:12,091 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,091 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:12,091 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,091 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:12,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:12,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:12,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c9335a4[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:12,094 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:12,094 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,095 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,095 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:12,095 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:12,095 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:12,095 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,095 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:12,096 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,096 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:12,097 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:12,097 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:12,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:12,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:12,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:12,099 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:12,099 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:12,099 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:12,099 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:12,099 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:12,102 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,102 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,102 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:12,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:12,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:12,102 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:12,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,104 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:12,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:12,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,106 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:12,106 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:12,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:12,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:12,106 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,106 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,106 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,106 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,106 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:12,106 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:12,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:12,107 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,107 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:12,107 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:12,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:12,107 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:12,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:12,183 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:12,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844232180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,183 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:12,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844232181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:12,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844232181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:12,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844232183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:12,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844232183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,207 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:12,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:12,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:12,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:12,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:12,211 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:12,211 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:12,211 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:12,211 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:12,211 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,211 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,215 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:12,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:12,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:12,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,215 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,215 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,215 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:12,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:12,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,216 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:12,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:12,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:12,216 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:12,216 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:12,216 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:12,216 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:12,216 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:12,216 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:12,216 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:12,216 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:12,216 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:12,217 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:12,217 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:12,217 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,217 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:12,217 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:12,220 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,220 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:12,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:12,220 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,220 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:12,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:12,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:12,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:12,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:12,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14c35db1[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:12,222 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:12,222 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,223 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,223 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:12,223 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:12,223 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,223 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:12,223 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:12,224 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,224 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:12,224 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:12,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:12,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:12,224 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:12,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:12,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:12,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:12,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:12,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:12,227 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:12,227 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:12,227 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:12,227 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:12,227 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:12,229 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,229 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:12,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,229 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:12,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:12,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:12,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:12,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:12,229 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:12,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,231 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:12,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,234 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,234 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,234 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,234 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:12,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:12,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:12,234 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:12,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:12,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:12,234 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:12,234 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,235 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:12,235 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:12,235 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,235 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,235 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:12,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:12,335 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:12,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:12,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:12,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:12,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:12,340 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:12,340 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:12,340 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:12,340 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:12,341 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,341 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,342 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,342 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:12,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:12,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:12,343 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,343 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:12,343 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:12,343 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:12,344 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:12,344 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:12,344 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:12,344 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:12,344 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:12,344 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:12,344 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:12,344 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:12,344 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:12,344 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:12,344 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:12,344 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,344 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:12,344 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:12,345 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,345 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:12,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:12,346 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,346 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:12,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:12,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:12,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:12,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:12,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@509c04c3[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:12,347 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:12,348 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,348 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,349 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:12,349 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:12,349 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:12,349 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,349 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:12,351 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,351 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:12,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:12,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:12,351 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:12,351 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:12,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:12,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:12,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:12,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:12,354 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:12,354 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:12,354 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:12,354 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:12,354 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:12,355 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,355 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:12,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:12,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,356 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:12,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:12,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:12,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:12,356 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:12,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,358 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:12,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,360 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:12,360 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:12,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:12,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:12,360 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,360 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,360 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,360 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,360 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,360 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:12,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:12,360 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:12,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:12,360 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:12,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:12,361 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:12,361 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:12,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:12,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844232384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:12,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844232385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:12,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844232385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:12,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844232386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:12,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844232386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,461 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:12,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:12,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:12,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:12,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:12,465 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:12,465 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:12,465 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:12,465 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:12,465 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,465 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,467 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:12,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:12,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:12,467 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,467 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,468 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:12,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,468 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:12,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:12,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:12,469 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:12,469 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:12,469 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:12,469 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:12,469 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:12,469 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:12,469 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:12,469 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:12,469 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:12,469 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:12,469 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:12,469 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,469 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:12,469 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:12,471 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418f854db00e5e341d8bc87d8c77b135510 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f854db00e5e341d8bc87d8c77b135510 2023-04-18 18:56:12,472 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:12,473 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:12,474 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,474 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:12,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:12,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@148cd133[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:12,475 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:12,475 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,476 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/0ca335a3162e4c738b1372c1da1aa001 2023-04-18 18:56:12,476 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,477 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:12,477 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:12,477 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:12,477 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,477 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:12,478 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:12,478 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:12,478 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:12,478 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:12,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:12,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:12,482 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:12,482 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:12,482 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:12,482 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:12,482 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:12,484 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:12,484 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,484 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:12,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:12,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:12,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:12,485 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/a2c42639e660462ea1cdf242afc36547 2023-04-18 18:56:12,485 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:12,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,487 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:12,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,493 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,493 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:12,493 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:12,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:12,493 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,493 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:12,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:12,493 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:12,494 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:12,494 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:12,494 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,494 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,494 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:12,493 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:12,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:12,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:12,594 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:12,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:12,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:12,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:12,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:12,597 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:12,597 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:12,597 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:12,597 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:12,598 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,598 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,599 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:12,599 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:12,600 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,600 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:12,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:12,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:12,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:12,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:12,601 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:12,601 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:12,601 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:12,601 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:12,601 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:12,601 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:12,601 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:12,601 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:12,601 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:12,601 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:12,601 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:12,601 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,601 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:12,601 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:12,605 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,605 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:12,606 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,606 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:12,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:12,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2187f813[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:12,607 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:12,607 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,610 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,610 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:12,610 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:12,610 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:12,610 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,610 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:12,612 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:12,612 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:12,612 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:12,612 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:12,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:12,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:12,614 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:12,615 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:12,615 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:12,615 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:12,615 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:12,616 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,616 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:12,616 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:12,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:12,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:12,617 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:12,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,618 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:12,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,620 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,620 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:12,620 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,620 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:12,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:12,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:12,620 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,621 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,621 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,621 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,621 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:12,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:12,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:12,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:12,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:12,621 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:12,621 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:12,621 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:12,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:12,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844232687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:12,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844232689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:12,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844232689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:12,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844232689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:12,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844232691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,722 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:12,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:12,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:12,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:12,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:12,726 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:12,726 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:12,726 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:12,726 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:12,726 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,726 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,729 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:12,729 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:12,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:12,729 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,729 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:12,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:12,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:12,730 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:12,730 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:12,730 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:12,731 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:12,731 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:12,731 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:12,731 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:12,731 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:12,731 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:12,731 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:12,731 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:12,731 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,731 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:12,731 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:12,733 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:12,733 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:12,734 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,734 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:12,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:12,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fa70747[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:12,735 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:12,735 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,736 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,736 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:12,736 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:12,736 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:12,736 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,736 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:12,737 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:12,737 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:12,737 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:12,737 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:12,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:12,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:12,740 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:12,740 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:12,740 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:12,740 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:12,740 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:12,743 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,743 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:12,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,743 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:12,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:12,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:12,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:12,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:12,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,744 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:12,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,746 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,748 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,748 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,748 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:12,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:12,749 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:12,749 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:12,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:12,748 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:12,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:12,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:12,749 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,749 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:12,749 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:12,749 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:12,749 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:12,749 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,849 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:12,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:12,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:12,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:12,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:12,853 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:12,853 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:12,853 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:12,853 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:12,853 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,853 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,855 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:12,855 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:12,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:12,855 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,855 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:12,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:12,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:12,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:12,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:12,856 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:12,856 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:12,856 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:12,857 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:12,857 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:12,857 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:12,857 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:12,857 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:12,857 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:12,857 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:12,857 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:12,857 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,857 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:12,857 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:12,858 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,858 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:12,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:12,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:12,859 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,859 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:12,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:12,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:12,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:12,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fada46c[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:12,860 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:12,860 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,862 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,862 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:12,862 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:12,862 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,863 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:12,862 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:12,864 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:12,864 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:12,864 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:12,864 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:12,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:12,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:12,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:12,866 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:12,866 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:12,866 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:12,867 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:12,867 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:12,868 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,868 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:12,868 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:12,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:12,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:12,869 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:12,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,870 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,872 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:12,872 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,872 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:12,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:12,872 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:12,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:12,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:12,872 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,872 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:12,873 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:12,873 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:12,872 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,873 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,873 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:12,873 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:12,895 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/4b03ad08bfeb43788d8d757c39d178f1 2023-04-18 18:56:12,899 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/0ca335a3162e4c738b1372c1da1aa001 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/0ca335a3162e4c738b1372c1da1aa001 2023-04-18 18:56:12,904 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/0ca335a3162e4c738b1372c1da1aa001, entries=150, sequenceid=51, filesize=25.1 K 2023-04-18 18:56:12,905 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/a2c42639e660462ea1cdf242afc36547 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/a2c42639e660462ea1cdf242afc36547 2023-04-18 18:56:12,908 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/a2c42639e660462ea1cdf242afc36547, entries=150, sequenceid=51, filesize=11.6 K 2023-04-18 18:56:12,908 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/4b03ad08bfeb43788d8d757c39d178f1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4b03ad08bfeb43788d8d757c39d178f1 2023-04-18 18:56:12,912 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4b03ad08bfeb43788d8d757c39d178f1, entries=150, sequenceid=51, filesize=11.6 K 2023-04-18 18:56:12,916 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=120.76 KB/123660 for c48561c851d33e29da879cafb393a8fd in 863ms, sequenceid=51, compaction requested=true 2023-04-18 18:56:12,917 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:12,917 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:12,917 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:12,917 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:12,917 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:12,917 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:12,925 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91118 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:12,925 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/A is initiating minor compaction (all files) 2023-04-18 18:56:12,925 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/A in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:12,925 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/9e367509f9094d9d989c77c04f1f4829, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/f9153095797c430fab62ca852f6ad1f7, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/0ca335a3162e4c738b1372c1da1aa001] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=89.0 K 2023-04-18 18:56:12,926 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:12,926 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/B is initiating minor compaction (all files) 2023-04-18 18:56:12,926 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/B in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:12,926 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/0eff99bcdbd746f687208d0a8061f09e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/59d6a9882f194717872cc7827af5e18b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/a2c42639e660462ea1cdf242afc36547] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=34.8 K 2023-04-18 18:56:12,926 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 9e367509f9094d9d989c77c04f1f4829, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681844171813 2023-04-18 18:56:12,927 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 0eff99bcdbd746f687208d0a8061f09e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681844171814 2023-04-18 18:56:12,927 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting f9153095797c430fab62ca852f6ad1f7, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=40, earliestPutTs=1681844171829 2023-04-18 18:56:12,927 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 59d6a9882f194717872cc7827af5e18b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=40, earliestPutTs=1681844171829 2023-04-18 18:56:12,928 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting a2c42639e660462ea1cdf242afc36547, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681844171946 2023-04-18 18:56:12,928 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 0ca335a3162e4c738b1372c1da1aa001, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681844171946 2023-04-18 18:56:12,941 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#B#compaction#555 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:12,945 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#A#compaction#554 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:12,973 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:12,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:12,974 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/71c05c954d8346dfa0f12267381e8952 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/71c05c954d8346dfa0f12267381e8952 2023-04-18 18:56:12,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:12,979 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c48561c851d33e29da879cafb393a8fd/B of c48561c851d33e29da879cafb393a8fd into 71c05c954d8346dfa0f12267381e8952(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:12,979 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:12,979 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/B, priority=13, startTime=1681844172917; duration=0sec 2023-04-18 18:56:12,979 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:12,979 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:12,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:12,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:12,980 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:12,980 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:12,980 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:12,980 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:12,980 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:12,980 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/C is initiating minor compaction (all files) 2023-04-18 18:56:12,981 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/C in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:12,981 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9c17dc07c85c48bf82ec6b1816214534, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/c592bb64f5fc420aa49be513f17e4ef5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4b03ad08bfeb43788d8d757c39d178f1] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=34.8 K 2023-04-18 18:56:12,981 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,981 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,981 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 9c17dc07c85c48bf82ec6b1816214534, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681844171814 2023-04-18 18:56:12,981 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting c592bb64f5fc420aa49be513f17e4ef5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=40, earliestPutTs=1681844171829 2023-04-18 18:56:12,982 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 4b03ad08bfeb43788d8d757c39d178f1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681844171946 2023-04-18 18:56:12,983 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:12,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:12,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:12,983 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,984 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:12,984 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:12,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:12,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:12,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:12,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:12,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:12,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:12,988 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:12,988 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:12,989 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:13,004 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#C#compaction#556 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:13,008 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:13,008 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:13,008 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:13,008 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:13,009 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=120.76 KB heapSize=351.95 KB 2023-04-18 18:56:13,009 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:13,009 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:13,009 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:13,010 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:13,010 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:13,010 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:13,013 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/6d0233c06fd544ad8d9d6eb1a5095b10 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/6d0233c06fd544ad8d9d6eb1a5095b10 2023-04-18 18:56:13,021 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c48561c851d33e29da879cafb393a8fd/A of c48561c851d33e29da879cafb393a8fd into 6d0233c06fd544ad8d9d6eb1a5095b10(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:13,021 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:13,022 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/A, priority=13, startTime=1681844172917; duration=0sec 2023-04-18 18:56:13,022 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:13,049 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418c46faac594624f6eacd2edf4ae42b027 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418c46faac594624f6eacd2edf4ae42b027 2023-04-18 18:56:13,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:13,193 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:13,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:13,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844233198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:13,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844233198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:13,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844233199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:13,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844233201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:13,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844233202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,227 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-18 18:56:13,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:13,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844233303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:13,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844233303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,308 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:13,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844233306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,308 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:13,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844233306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,430 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/5d1115953ab64140a00444ef9d0f7d68 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/5d1115953ab64140a00444ef9d0f7d68 2023-04-18 18:56:13,433 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c48561c851d33e29da879cafb393a8fd/C of c48561c851d33e29da879cafb393a8fd into 5d1115953ab64140a00444ef9d0f7d68(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:13,433 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:13,433 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/C, priority=13, startTime=1681844172917; duration=0sec 2023-04-18 18:56:13,433 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:13,456 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=77, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/f8e10ad267914a1b94cd552c9e7668c8 2023-04-18 18:56:13,482 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=77 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/9c1ea6a8d6194fe0a06d94582cf52918 2023-04-18 18:56:13,493 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=77 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/9c330e9ad72b43648a8c79ddbb89f661 2023-04-18 18:56:13,504 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/f8e10ad267914a1b94cd552c9e7668c8 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/f8e10ad267914a1b94cd552c9e7668c8 2023-04-18 18:56:13,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:13,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844233505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:13,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844233507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,509 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/f8e10ad267914a1b94cd552c9e7668c8, entries=150, sequenceid=77, filesize=25.1 K 2023-04-18 18:56:13,510 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/9c1ea6a8d6194fe0a06d94582cf52918 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/9c1ea6a8d6194fe0a06d94582cf52918 2023-04-18 18:56:13,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:13,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:13,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844233509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844233509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,514 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/9c1ea6a8d6194fe0a06d94582cf52918, entries=150, sequenceid=77, filesize=11.6 K 2023-04-18 18:56:13,528 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/9c330e9ad72b43648a8c79ddbb89f661 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9c330e9ad72b43648a8c79ddbb89f661 2023-04-18 18:56:13,533 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9c330e9ad72b43648a8c79ddbb89f661, entries=150, sequenceid=77, filesize=11.6 K 2023-04-18 18:56:13,534 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~351.91 KB/360352, currentSize=53.67 KB/54960 for c48561c851d33e29da879cafb393a8fd in 525ms, sequenceid=77, compaction requested=false 2023-04-18 18:56:13,534 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:13,534 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:13,534 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:13,534 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:13,534 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,539 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:13,539 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:13,540 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,541 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:13,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:13,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:13,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:13,542 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:13,542 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:13,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:13,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:13,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ddafbe3[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:13,545 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:13,545 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:13,546 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:13,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:13,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:13,547 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,547 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:13,551 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:13,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:13,551 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:13,551 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:13,551 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:13,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:13,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:13,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:13,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:13,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:13,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:13,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:13,554 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:13,554 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:13,554 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:13,554 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:13,554 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:13,555 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:13,555 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:13,555 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:13,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:13,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:13,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:13,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:13,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:13,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:13,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:13,556 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:13,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:13,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:13,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:13,557 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:13,559 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:13,559 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:13,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:13,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:13,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:13,560 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,560 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:13,560 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:13,560 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,560 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:13,560 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:13,560 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:13,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:13,560 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:13,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:13,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:13,560 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:13,560 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:13,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:13,660 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:13,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:13,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:13,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:13,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:13,664 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:13,664 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:13,664 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:13,664 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:13,664 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:13,664 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:13,665 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:13,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:13,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:13,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:13,666 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:13,667 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,667 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:13,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:13,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:13,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:13,667 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:13,667 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:13,667 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:13,667 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:13,667 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:13,667 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:13,667 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:13,667 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:13,667 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:13,668 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-18 18:56:13,668 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:13,668 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:13,668 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:13,668 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:13,668 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:13,668 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:13,683 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304181d60ca05b17043c386915f43698745c2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304181d60ca05b17043c386915f43698745c2 2023-04-18 18:56:13,689 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=90, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/c4336d4de34841ec924faf183203cd0d 2023-04-18 18:56:13,704 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=90 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/83ef252fdc094e348afc378dafb415f0 2023-04-18 18:56:13,716 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=90 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/24d197483e164a028e5c59c75d1b287c 2023-04-18 18:56:13,720 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/c4336d4de34841ec924faf183203cd0d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/c4336d4de34841ec924faf183203cd0d 2023-04-18 18:56:13,725 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/c4336d4de34841ec924faf183203cd0d, entries=150, sequenceid=90, filesize=25.1 K 2023-04-18 18:56:13,726 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/83ef252fdc094e348afc378dafb415f0 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/83ef252fdc094e348afc378dafb415f0 2023-04-18 18:56:13,729 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/83ef252fdc094e348afc378dafb415f0, entries=150, sequenceid=90, filesize=11.6 K 2023-04-18 18:56:13,730 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/24d197483e164a028e5c59c75d1b287c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/24d197483e164a028e5c59c75d1b287c 2023-04-18 18:56:13,740 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/24d197483e164a028e5c59c75d1b287c, entries=150, sequenceid=90, filesize=11.6 K 2023-04-18 18:56:13,741 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for c48561c851d33e29da879cafb393a8fd in 74ms, sequenceid=90, compaction requested=true 2023-04-18 18:56:13,741 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:13,741 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:13,741 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:13,742 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:13,742 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,742 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:13,742 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:13,744 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,744 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:13,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:13,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:13,744 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:13,745 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:13,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:13,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:13,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7383c12b[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:13,746 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:13,746 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:13,747 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:13,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:13,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:13,748 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:13,748 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:13,748 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:13,748 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,748 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:13,749 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,749 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:13,749 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:13,749 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:13,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:13,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:13,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:13,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:13,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:13,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:13,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:13,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:13,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:13,752 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:13,752 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:13,752 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:13,752 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:13,752 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:13,754 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:13,754 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:13,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:13,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:13,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:13,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:13,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:13,754 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:13,755 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:13,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:13,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:13,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:13,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:13,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:13,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:13,756 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:13,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:13,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,759 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:13,759 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:13,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:13,759 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:13,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:13,759 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:13,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:13,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:13,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:13,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:13,759 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:13,760 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:13,760 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:13,760 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:13,760 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,760 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:13,760 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:13,760 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:13,860 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:13,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:13,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:13,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:13,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:13,863 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:13,863 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:13,863 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:13,863 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:13,863 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:13,863 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:13,864 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:13,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:13,864 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:13,865 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,865 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:13,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:13,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:13,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:13,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:13,866 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:13,866 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:13,866 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:13,868 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:13,868 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:13,868 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:13,868 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:13,868 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-18 18:56:13,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:13,878 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:13,878 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:13,878 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:13,878 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:13,878 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:13,879 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:13,879 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:13,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:13,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844233899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844173896, max timestamp=1681844173902Num uniques 859; 2023-04-18 18:56:13,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:56:13,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844173896, max timestamp=1681844173902Num uniques 859; 2023-04-18 18:56:13,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:56:13,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844173896, max timestamp=1681844173902Num uniques 859; 2023-04-18 18:56:13,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:56:13,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:13,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844233920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:13,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844233922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:13,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:13,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844233923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:14,005 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:14,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844234004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:14,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:14,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844234025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:14,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:14,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844234026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:14,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:14,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844234026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:14,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:14,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844234203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:14,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:14,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844234206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:14,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:14,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844234229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:14,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:14,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844234229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:14,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:14,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844234229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:14,297 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304186aa61604ed5f470991952de2930c4172 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304186aa61604ed5f470991952de2930c4172 2023-04-18 18:56:14,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:14,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844234510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:14,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:14,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844234533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:14,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:14,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844234533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:14,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:14,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844234533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:14,703 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=102, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/586084c7f8454b808b2ca800c3547616 2023-04-18 18:56:14,716 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/b849c2de926940709d04ca702c728628 2023-04-18 18:56:15,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:15,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844235012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:15,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844235036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,040 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:15,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844235038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:15,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844235038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,126 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/65f50c4bf82149c7a23defc4fcc3b7c4 2023-04-18 18:56:15,130 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/586084c7f8454b808b2ca800c3547616 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/586084c7f8454b808b2ca800c3547616 2023-04-18 18:56:15,134 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/586084c7f8454b808b2ca800c3547616, entries=150, sequenceid=102, filesize=25.1 K 2023-04-18 18:56:15,134 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/b849c2de926940709d04ca702c728628 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b849c2de926940709d04ca702c728628 2023-04-18 18:56:15,138 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b849c2de926940709d04ca702c728628, entries=150, sequenceid=102, filesize=11.6 K 2023-04-18 18:56:15,139 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/65f50c4bf82149c7a23defc4fcc3b7c4 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/65f50c4bf82149c7a23defc4fcc3b7c4 2023-04-18 18:56:15,142 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/65f50c4bf82149c7a23defc4fcc3b7c4, entries=150, sequenceid=102, filesize=11.6 K 2023-04-18 18:56:15,143 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for c48561c851d33e29da879cafb393a8fd in 1275ms, sequenceid=102, compaction requested=true 2023-04-18 18:56:15,143 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:15,143 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:15,143 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:15,143 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:15,143 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,143 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:15,143 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:15,144 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,144 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:15,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:15,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:15,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:15,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:15,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:15,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:15,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:15,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64097d23[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:15,147 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:15,147 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,148 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:15,148 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,149 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:15,151 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,151 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:15,151 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:15,151 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:15,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:15,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:15,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:15,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:15,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:15,154 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:15,154 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:15,154 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:15,154 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:15,154 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:15,155 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,155 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:15,155 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:15,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:15,156 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:15,157 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:15,160 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:15,160 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:15,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:15,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:15,160 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,160 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:15,160 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:15,160 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,160 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,160 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,160 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:15,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:15,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:15,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:15,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:15,160 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:15,161 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:15,161 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:15,261 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:15,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:15,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:15,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:15,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:15,264 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:15,264 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:15,265 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:15,265 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:15,265 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,265 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:15,266 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:15,266 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:15,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:15,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:15,267 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,267 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:15,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:15,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:15,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:15,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:15,268 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:15,268 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:15,268 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:15,268 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:15,268 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:15,268 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:15,268 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:15,268 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-18 18:56:15,268 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:15,269 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:15,269 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:15,269 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:15,269 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:15,269 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:15,286 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418e10bf052840047338dffa4870c839c34 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e10bf052840047338dffa4870c839c34 2023-04-18 18:56:15,297 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=136, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/ad3dc098af214a6da75afd4ff3caa85b 2023-04-18 18:56:15,707 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/cfbcd3f910dd4d9e8bdff8797dd53044 2023-04-18 18:56:15,719 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/4079798575dd46b08e8e44e691d8b6ef 2023-04-18 18:56:15,724 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/ad3dc098af214a6da75afd4ff3caa85b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/ad3dc098af214a6da75afd4ff3caa85b 2023-04-18 18:56:15,728 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/ad3dc098af214a6da75afd4ff3caa85b, entries=150, sequenceid=136, filesize=25.3 K 2023-04-18 18:56:15,728 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/cfbcd3f910dd4d9e8bdff8797dd53044 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/cfbcd3f910dd4d9e8bdff8797dd53044 2023-04-18 18:56:15,732 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/cfbcd3f910dd4d9e8bdff8797dd53044, entries=150, sequenceid=136, filesize=11.8 K 2023-04-18 18:56:15,732 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/4079798575dd46b08e8e44e691d8b6ef as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4079798575dd46b08e8e44e691d8b6ef 2023-04-18 18:56:15,736 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4079798575dd46b08e8e44e691d8b6ef, entries=150, sequenceid=136, filesize=11.8 K 2023-04-18 18:56:15,736 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=0 B/0 for c48561c851d33e29da879cafb393a8fd in 468ms, sequenceid=136, compaction requested=true 2023-04-18 18:56:15,736 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:15,736 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:15,736 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:15,737 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:15,737 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,737 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:15,737 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:15,739 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:15,739 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:15,740 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,740 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:15,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:15,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:15,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:15,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:15,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:15,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16752b0a[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:15,741 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:15,741 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,743 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,743 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:15,743 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:15,743 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:15,743 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,743 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:15,744 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,744 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:15,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:15,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:15,744 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:15,745 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:15,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:15,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:15,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:15,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:15,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:15,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:15,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:15,747 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:15,747 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:15,747 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:15,747 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:15,747 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:15,753 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,753 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,753 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:15,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:15,754 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:15,755 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:15,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:15,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,758 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:15,758 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:15,758 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,758 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:15,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:15,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:15,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:15,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:15,758 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:15,758 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:15,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:15,758 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:15,758 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:15,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:15,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:15,758 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,759 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,759 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,759 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,858 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:15,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:15,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:15,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:15,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:15,862 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:15,863 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:15,863 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:15,863 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:15,863 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,863 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:15,864 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:15,864 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:15,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:15,865 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,865 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:15,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:15,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:15,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:15,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:15,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:15,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:15,866 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:15,866 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:15,866 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:15,866 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:15,866 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:15,866 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:15,866 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:15,867 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:15,867 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:15,867 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:15,867 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:15,867 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,867 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:15,867 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:15,868 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,868 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:15,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:15,869 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,869 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:15,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:15,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:15,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:15,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c4acfc4[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:15,871 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:15,871 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,873 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,874 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:15,874 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:15,874 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:15,874 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,874 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:15,875 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,875 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:15,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:15,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:15,875 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:15,875 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:15,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:15,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:15,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:15,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:15,878 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:15,878 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:15,878 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:15,878 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:15,878 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:15,879 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,879 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:15,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:15,879 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:15,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:15,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:15,880 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:15,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:15,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:15,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:15,882 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:15,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:15,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,884 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,884 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:15,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:15,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:15,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:15,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:15,884 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:15,885 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:15,885 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:15,884 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:15,884 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:15,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:15,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:15,885 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:15,885 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,885 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,885 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,885 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:15,985 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:15,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:15,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:15,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:15,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:15,989 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:15,989 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:15,989 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:15,989 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:15,989 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,989 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:15,992 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,992 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:15,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:15,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:15,993 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,993 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:15,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:15,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:15,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:15,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:15,994 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:15,994 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:15,994 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:15,994 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:15,994 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:15,994 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:15,994 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:15,995 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:15,995 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:15,995 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:15,995 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:15,995 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,995 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:15,995 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:15,996 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,996 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:15,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:15,997 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,997 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:15,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:15,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:15,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:15,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:15,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:15,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:15,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@412dd9ff[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:15,998 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:15,998 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:15,999 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,000 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:16,000 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:16,000 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:16,000 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,000 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:16,001 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,001 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:16,001 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:16,001 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:16,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:16,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:16,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:16,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:16,004 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:16,004 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:16,004 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:16,004 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:16,004 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:16,007 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,007 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:16,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,007 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:16,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:16,008 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:16,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,010 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:16,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,012 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,012 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:16,012 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:16,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:16,012 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,012 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,012 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:16,012 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:16,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:16,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:16,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:16,012 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:16,012 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,013 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,013 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:16,013 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:16,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:16,068 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 18:56:16,069 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:16,069 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:16,069 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:16,069 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:16,069 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:16,069 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:16,085 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304186c8f0fa98bd846a1b2efa160ee5c2a65 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304186c8f0fa98bd846a1b2efa160ee5c2a65 2023-04-18 18:56:16,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844176092, max timestamp=1681844176098Num uniques 859; 2023-04-18 18:56:16,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:56:16,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844176092, max timestamp=1681844176098Num uniques 859; 2023-04-18 18:56:16,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:56:16,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844176092, max timestamp=1681844176098Num uniques 859; 2023-04-18 18:56:16,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:56:16,113 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:16,114 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=149, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/974e5156e04b44d092854b05711126e4 2023-04-18 18:56:16,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:16,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:16,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:16,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:16,124 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:16,124 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:16,124 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:16,124 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:16,125 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,125 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,126 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:16,126 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:16,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:16,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,127 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,127 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:16,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:16,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:16,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:16,133 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:16,133 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:16,133 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:16,133 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:16,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:16,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844236130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,136 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:16,136 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:16,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:16,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844236131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:16,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844236132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,136 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:16,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:16,136 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:16,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844236133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,137 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:16,137 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:16,137 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:16,137 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:16,137 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,142 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:16,142 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:16,144 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,144 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:16,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:16,144 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,144 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:16,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:16,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:16,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:16,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:16,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e62449c[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:16,147 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:16,147 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,149 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,149 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:16,149 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:16,149 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:16,150 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,150 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:16,151 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:16,151 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,151 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:16,151 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:16,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:16,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:16,154 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:16,154 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:16,154 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:16,154 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:16,154 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:16,156 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,156 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:16,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,156 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:16,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:16,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:16,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:16,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:16,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,157 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,158 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,163 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,163 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,163 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:16,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:16,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:16,163 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:16,163 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,164 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,164 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,164 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,164 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:16,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:16,163 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:16,164 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:16,164 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:16,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:16,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:16,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844236223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,227 DEBUG [Thread-3981] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:56:16,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:16,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844236237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:16,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844236237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:16,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844236237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:16,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844236237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,264 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:16,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:16,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:16,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:16,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:16,268 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:16,268 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:16,268 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:16,268 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:16,269 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,269 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,270 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,270 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:16,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:16,270 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,270 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:16,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:16,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:16,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:16,272 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:16,272 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:16,272 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:16,272 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:16,272 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:16,272 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:16,272 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:16,272 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:16,272 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:16,272 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:16,272 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:16,272 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,272 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:16,272 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:16,274 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:16,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:16,274 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:16,274 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,274 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:16,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:16,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:16,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51f01f21[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:16,275 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:16,276 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,277 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,277 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,277 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:16,277 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:16,278 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:16,278 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:16,280 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,280 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:16,280 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:16,280 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:16,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:16,283 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:16,283 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:16,283 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:16,283 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:16,283 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:16,284 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,284 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:16,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,285 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:16,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:16,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:16,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:16,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:16,285 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:16,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,286 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:16,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,289 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:16,289 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:16,289 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,289 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,289 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:16,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:16,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:16,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:16,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:16,289 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,289 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:16,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:16,289 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:16,289 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,289 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,289 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:16,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:16,290 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,389 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:16,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:16,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:16,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:16,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:16,393 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:16,393 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:16,394 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:16,394 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:16,394 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,394 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,395 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:16,395 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:16,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:16,396 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,396 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:16,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:16,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:16,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:16,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:16,397 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:16,397 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:16,397 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:16,397 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:16,397 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:16,397 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:16,397 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:16,398 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:16,398 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:16,398 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:16,398 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:16,398 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,398 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:16,398 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:16,401 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,401 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:16,401 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,401 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:16,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:16,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:16,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:16,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bbede34[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:16,403 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:16,403 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,404 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,404 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:16,404 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:16,404 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:16,404 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,405 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:16,406 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,406 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:16,406 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:16,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:16,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:16,406 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:16,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:16,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:16,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:16,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:16,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:16,409 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:16,409 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:16,409 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:16,409 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:16,409 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:16,410 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,410 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,410 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:16,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:16,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:16,411 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:16,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:16,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:16,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:16,412 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:16,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,415 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,415 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:16,415 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:16,415 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,415 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,415 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,416 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,416 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,416 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:16,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:16,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:16,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:16,415 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:16,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:16,416 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:16,416 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:16,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:16,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844236440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:16,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844236440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:16,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844236442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:16,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844236442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,516 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:16,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:16,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:16,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:16,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:16,520 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:16,520 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:16,520 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:16,520 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:16,520 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,520 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,522 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:16,522 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:16,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:16,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,522 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,522 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:16,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:16,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:16,523 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:16,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:16,523 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:16,523 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:16,523 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:16,523 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:16,523 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:16,524 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:16,524 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:16,524 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:16,524 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:16,524 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:16,524 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:16,524 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,524 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:16,524 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:16,525 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,525 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:16,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:16,526 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,526 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:16,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:16,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:16,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:16,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:16,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f3089d[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:16,528 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:16,528 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,529 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,529 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:16,529 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:16,529 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:16,530 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,530 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:16,532 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,533 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:16,533 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:16,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:16,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:16,533 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:16,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:16,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:16,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:16,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:16,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:16,536 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:16,536 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:16,536 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:16,536 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:16,536 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:16,538 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,538 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:16,538 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:16,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:16,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,538 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:16,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:16,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:16,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:16,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,539 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:16,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:16,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,545 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,546 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,546 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,546 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,546 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:16,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:16,546 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:16,546 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,546 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:16,546 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:16,546 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:16,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:16,546 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/fa7b440a161946efb195b6e2cf07dbb3 2023-04-18 18:56:16,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:16,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:16,547 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:16,547 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:16,557 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/241323bdc5704938bc1c0bda5b86db8a 2023-04-18 18:56:16,560 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/974e5156e04b44d092854b05711126e4 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/974e5156e04b44d092854b05711126e4 2023-04-18 18:56:16,564 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/974e5156e04b44d092854b05711126e4, entries=350, sequenceid=149, filesize=52.6 K 2023-04-18 18:56:16,565 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/fa7b440a161946efb195b6e2cf07dbb3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fa7b440a161946efb195b6e2cf07dbb3 2023-04-18 18:56:16,569 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fa7b440a161946efb195b6e2cf07dbb3, entries=150, sequenceid=149, filesize=11.8 K 2023-04-18 18:56:16,570 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/241323bdc5704938bc1c0bda5b86db8a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/241323bdc5704938bc1c0bda5b86db8a 2023-04-18 18:56:16,574 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/241323bdc5704938bc1c0bda5b86db8a, entries=150, sequenceid=149, filesize=11.8 K 2023-04-18 18:56:16,575 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for c48561c851d33e29da879cafb393a8fd in 507ms, sequenceid=149, compaction requested=true 2023-04-18 18:56:16,575 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:16,575 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:16,575 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:16,575 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-18 18:56:16,575 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-18 18:56:16,575 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:16,577 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 182843 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-18 18:56:16,577 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/A is initiating minor compaction (all files) 2023-04-18 18:56:16,577 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/A in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:16,577 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/6d0233c06fd544ad8d9d6eb1a5095b10, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/f8e10ad267914a1b94cd552c9e7668c8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/c4336d4de34841ec924faf183203cd0d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/586084c7f8454b808b2ca800c3547616, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/ad3dc098af214a6da75afd4ff3caa85b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/974e5156e04b44d092854b05711126e4] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=178.6 K 2023-04-18 18:56:16,577 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71719 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-18 18:56:16,577 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/B is initiating minor compaction (all files) 2023-04-18 18:56:16,577 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/B in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:16,577 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/71c05c954d8346dfa0f12267381e8952, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/9c1ea6a8d6194fe0a06d94582cf52918, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/83ef252fdc094e348afc378dafb415f0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b849c2de926940709d04ca702c728628, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/cfbcd3f910dd4d9e8bdff8797dd53044, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fa7b440a161946efb195b6e2cf07dbb3] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=70.0 K 2023-04-18 18:56:16,577 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 6d0233c06fd544ad8d9d6eb1a5095b10, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681844171946 2023-04-18 18:56:16,578 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 71c05c954d8346dfa0f12267381e8952, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681844171946 2023-04-18 18:56:16,578 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting f8e10ad267914a1b94cd552c9e7668c8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1681844172073 2023-04-18 18:56:16,578 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 9c1ea6a8d6194fe0a06d94582cf52918, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1681844172073 2023-04-18 18:56:16,578 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting c4336d4de34841ec924faf183203cd0d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=90, earliestPutTs=1681844173193 2023-04-18 18:56:16,578 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 83ef252fdc094e348afc378dafb415f0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=90, earliestPutTs=1681844173193 2023-04-18 18:56:16,578 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 586084c7f8454b808b2ca800c3547616, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681844173837 2023-04-18 18:56:16,579 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b849c2de926940709d04ca702c728628, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681844173837 2023-04-18 18:56:16,579 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting ad3dc098af214a6da75afd4ff3caa85b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1681844173919 2023-04-18 18:56:16,579 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting cfbcd3f910dd4d9e8bdff8797dd53044, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1681844173919 2023-04-18 18:56:16,579 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 974e5156e04b44d092854b05711126e4, keycount=350, bloomtype=ROW, size=52.6 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681844176038 2023-04-18 18:56:16,579 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting fa7b440a161946efb195b6e2cf07dbb3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681844176061 2023-04-18 18:56:16,589 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#B#compaction#573 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:16,594 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#A#compaction#572 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:16,647 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:16,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:16,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:16,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:16,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:16,650 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:16,650 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:16,650 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:16,650 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:16,651 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,651 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,652 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:16,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:16,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:16,652 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,653 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,653 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:16,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:16,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:16,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:16,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:16,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:16,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:16,654 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:16,654 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:16,654 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:16,654 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:16,654 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:16,654 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:16,654 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:16,654 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-18 18:56:16,654 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:16,654 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:16,654 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:16,654 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:16,655 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:16,655 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:16,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:16,748 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:16,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:16,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844236753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:16,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844236754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:16,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844236756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:16,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844236757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:16,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844236858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:16,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844236859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:16,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844236862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:16,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:16,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844236863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,004 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/b970553f344c4953b30a213313232a6c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b970553f344c4953b30a213313232a6c 2023-04-18 18:56:17,008 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in c48561c851d33e29da879cafb393a8fd/B of c48561c851d33e29da879cafb393a8fd into b970553f344c4953b30a213313232a6c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:17,008 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:17,008 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/B, priority=10, startTime=1681844176575; duration=0sec 2023-04-18 18:56:17,008 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:17,008 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-18 18:56:17,010 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71719 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-18 18:56:17,010 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/C is initiating minor compaction (all files) 2023-04-18 18:56:17,010 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/C in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:17,010 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/5d1115953ab64140a00444ef9d0f7d68, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9c330e9ad72b43648a8c79ddbb89f661, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/24d197483e164a028e5c59c75d1b287c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/65f50c4bf82149c7a23defc4fcc3b7c4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4079798575dd46b08e8e44e691d8b6ef, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/241323bdc5704938bc1c0bda5b86db8a] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=70.0 K 2023-04-18 18:56:17,010 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 5d1115953ab64140a00444ef9d0f7d68, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681844171946 2023-04-18 18:56:17,011 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 9c330e9ad72b43648a8c79ddbb89f661, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1681844172073 2023-04-18 18:56:17,011 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 24d197483e164a028e5c59c75d1b287c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=90, earliestPutTs=1681844173193 2023-04-18 18:56:17,012 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 65f50c4bf82149c7a23defc4fcc3b7c4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681844173837 2023-04-18 18:56:17,012 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 4079798575dd46b08e8e44e691d8b6ef, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1681844173919 2023-04-18 18:56:17,012 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 241323bdc5704938bc1c0bda5b86db8a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681844176061 2023-04-18 18:56:17,016 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/89f97c0dc85c4c4195e82a1b9b141337 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/89f97c0dc85c4c4195e82a1b9b141337 2023-04-18 18:56:17,024 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in c48561c851d33e29da879cafb393a8fd/A of c48561c851d33e29da879cafb393a8fd into 89f97c0dc85c4c4195e82a1b9b141337(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:17,024 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:17,024 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/A, priority=10, startTime=1681844176575; duration=0sec 2023-04-18 18:56:17,024 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:17,026 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#C#compaction#575 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:17,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:17,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844237062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:17,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844237062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:17,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844237066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:17,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844237066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,076 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418263e36bec82849489de162f57a4945e1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418263e36bec82849489de162f57a4945e1 2023-04-18 18:56:17,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:17,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844237366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:17,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844237367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:17,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844237370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:17,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844237370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,435 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/f14ee3a998be4ca89f736e1fa6adde27 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/f14ee3a998be4ca89f736e1fa6adde27 2023-04-18 18:56:17,439 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in c48561c851d33e29da879cafb393a8fd/C of c48561c851d33e29da879cafb393a8fd into f14ee3a998be4ca89f736e1fa6adde27(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:17,439 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:17,439 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/C, priority=10, startTime=1681844176575; duration=0sec 2023-04-18 18:56:17,439 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:17,482 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=185, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/bb4b45fa96664dadaa72ffb577b9bd40 2023-04-18 18:56:17,493 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/3302e82a60964e80a8ac241c16c32f2f 2023-04-18 18:56:17,504 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/b2d9d644ae7e4b6a8820dc348d1c173a 2023-04-18 18:56:17,510 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/bb4b45fa96664dadaa72ffb577b9bd40 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/bb4b45fa96664dadaa72ffb577b9bd40 2023-04-18 18:56:17,514 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/bb4b45fa96664dadaa72ffb577b9bd40, entries=150, sequenceid=185, filesize=25.3 K 2023-04-18 18:56:17,515 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/3302e82a60964e80a8ac241c16c32f2f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/3302e82a60964e80a8ac241c16c32f2f 2023-04-18 18:56:17,519 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/3302e82a60964e80a8ac241c16c32f2f, entries=150, sequenceid=185, filesize=11.8 K 2023-04-18 18:56:17,520 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/b2d9d644ae7e4b6a8820dc348d1c173a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/b2d9d644ae7e4b6a8820dc348d1c173a 2023-04-18 18:56:17,522 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/b2d9d644ae7e4b6a8820dc348d1c173a, entries=150, sequenceid=185, filesize=11.8 K 2023-04-18 18:56:17,523 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for c48561c851d33e29da879cafb393a8fd in 869ms, sequenceid=185, compaction requested=false 2023-04-18 18:56:17,523 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:17,523 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:17,523 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:17,523 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:17,523 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,523 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:17,523 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:17,525 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,525 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:17,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:17,526 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,526 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:17,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:17,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:17,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:17,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:17,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78c9ea3b[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:17,529 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:17,529 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,531 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,531 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,531 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:17,531 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:17,531 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:17,531 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:17,534 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:17,534 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:17,534 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:17,536 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:17,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:17,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:17,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:17,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:17,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:17,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:17,540 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:17,540 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:17,540 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:17,540 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:17,540 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:17,543 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,543 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:17,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,543 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:17,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:17,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:17,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:17,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:17,544 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:17,546 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:17,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,549 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,550 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:17,550 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:17,550 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,550 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,550 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:17,550 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:17,550 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,550 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:17,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:17,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:17,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:17,550 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:17,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:17,550 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:17,550 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:17,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:17,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:17,650 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:17,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:17,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:17,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:17,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:17,654 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:17,654 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:17,654 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:17,654 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:17,654 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,654 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:17,655 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:17,655 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:17,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:17,656 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,656 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:17,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:17,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:17,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:17,659 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:17,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:17,660 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:17,660 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:17,660 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:17,660 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:17,660 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:17,660 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:17,660 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:17,660 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-18 18:56:17,660 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:17,660 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:17,660 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:17,661 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:17,661 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:17,661 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:17,679 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418c9e74a5139144b68b5c38ea04e85a19e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418c9e74a5139144b68b5c38ea04e85a19e 2023-04-18 18:56:17,684 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=199, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/2fc3aee7c71149d9b02450907900d77d 2023-04-18 18:56:17,694 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/4d124fd5e4ce413987828181719d8a05 2023-04-18 18:56:17,710 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/78a7ebbe6df749199cf84f04c9cccac3 2023-04-18 18:56:17,714 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/2fc3aee7c71149d9b02450907900d77d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/2fc3aee7c71149d9b02450907900d77d 2023-04-18 18:56:17,717 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/2fc3aee7c71149d9b02450907900d77d, entries=150, sequenceid=199, filesize=25.3 K 2023-04-18 18:56:17,718 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/4d124fd5e4ce413987828181719d8a05 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/4d124fd5e4ce413987828181719d8a05 2023-04-18 18:56:17,721 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/4d124fd5e4ce413987828181719d8a05, entries=150, sequenceid=199, filesize=11.8 K 2023-04-18 18:56:17,722 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/78a7ebbe6df749199cf84f04c9cccac3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/78a7ebbe6df749199cf84f04c9cccac3 2023-04-18 18:56:17,724 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/78a7ebbe6df749199cf84f04c9cccac3, entries=150, sequenceid=199, filesize=11.8 K 2023-04-18 18:56:17,725 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for c48561c851d33e29da879cafb393a8fd in 65ms, sequenceid=199, compaction requested=true 2023-04-18 18:56:17,725 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:17,725 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:17,725 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:17,725 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:17,725 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,725 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:17,725 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:17,728 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,728 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:17,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:17,728 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,728 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:17,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:17,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:17,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:17,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:17,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2991fb14[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:17,730 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:17,730 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,731 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,732 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:17,732 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:17,732 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,732 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:17,732 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:17,733 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:17,733 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:17,733 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:17,733 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:17,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:17,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:17,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:17,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:17,736 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:17,736 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:17,736 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:17,736 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:17,736 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:17,738 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,738 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,738 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:17,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:17,738 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:17,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,740 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:17,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,743 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:17,743 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,743 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:17,743 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:17,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:17,744 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:17,744 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:17,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:17,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:17,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:17,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:17,744 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,744 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,744 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,744 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,744 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:17,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:17,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:17,744 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:17,844 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:17,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:17,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:17,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:17,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:17,848 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:17,848 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:17,848 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:17,848 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:17,851 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,851 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:17,852 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,852 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:17,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:17,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:17,853 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,853 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:17,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:17,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:17,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:17,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:17,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:17,854 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:17,854 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:17,854 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:17,854 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:17,854 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:17,854 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:17,855 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:17,855 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:17,854 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:17,855 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:17,855 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:17,855 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,855 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:17,855 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:17,856 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,856 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:17,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:17,857 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,857 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:17,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:17,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:17,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:17,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:17,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7483cf6c[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:17,858 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:17,858 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,859 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,860 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:17,860 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:17,860 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:17,860 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,860 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:17,862 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,862 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:17,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:17,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:17,862 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:17,863 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:17,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:17,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:17,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:17,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:17,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:17,865 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:17,865 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:17,865 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:17,865 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:17,865 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:17,866 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,866 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:17,866 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:17,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:17,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:17,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:17,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:17,867 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,868 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:17,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:17,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:17,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:17,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:17,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:17,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:17,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,871 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:17,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:17,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:17,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,871 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:17,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:17,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:17,871 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:17,871 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:17,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:17,898 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 18:56:17,899 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:17,899 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:17,899 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:17,899 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:17,899 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:17,899 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:17,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:17,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:17,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844237924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844237924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844177921, max timestamp=1681844177926Num uniques 859; 2023-04-18 18:56:17,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:56:17,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844177921, max timestamp=1681844177926Num uniques 859; 2023-04-18 18:56:17,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844177921, max timestamp=1681844177926Num uniques 859; 2023-04-18 18:56:17,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:56:17,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:56:17,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:17,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844237954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:17,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844237956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,972 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:17,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:17,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:17,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:17,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:17,975 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:17,975 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:17,975 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:17,975 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:17,976 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,976 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:17,977 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:17,977 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:17,978 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,978 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:17,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:17,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:17,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:17,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:17,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:17,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:17,979 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:17,979 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:17,979 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:17,979 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:17,979 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:17,979 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:17,979 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:17,979 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:17,980 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:17,980 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:17,980 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:17,980 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,980 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:17,980 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:17,985 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,985 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:17,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:17,985 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,985 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:17,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:17,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:17,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:17,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:17,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68a89681[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:17,987 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:17,987 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,988 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:17,988 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:17,988 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:17,989 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:17,989 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,989 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:17,990 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:17,991 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:17,990 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,991 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:17,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:17,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:17,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:17,994 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:17,994 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:17,994 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:17,994 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:17,994 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:17,996 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,996 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,997 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:17,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:17,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:17,997 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:17,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:17,999 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:17,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:17,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:18,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,002 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,002 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,002 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,002 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:18,002 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:18,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:18,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:18,002 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,002 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:18,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:18,003 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:18,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:18,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:18,002 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,003 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:18,003 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:18,003 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:18,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:18,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844238028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:18,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844238029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:18,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844238058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:18,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844238061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,103 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:18,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:18,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:18,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:18,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:18,107 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:18,107 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:18,107 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:18,107 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:18,107 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,107 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,109 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:18,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:18,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:18,109 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,109 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,109 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:18,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:18,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:18,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:18,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:18,111 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:18,111 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:18,111 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:18,111 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:18,111 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:18,111 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:18,111 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:18,111 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:18,112 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:18,112 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:18,112 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:18,112 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,112 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:18,112 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:18,113 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,113 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:18,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:18,114 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,114 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:18,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:18,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:18,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:18,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:18,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f07219d[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:18,115 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:18,115 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,117 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,117 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:18,117 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:18,117 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:18,117 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,117 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:18,120 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:18,120 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:18,120 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,120 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:18,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:18,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:18,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:18,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:18,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:18,123 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:18,123 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:18,123 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:18,123 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:18,123 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:18,125 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,125 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,125 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:18,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,126 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,126 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:18,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:18,128 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:18,129 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,129 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:18,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:18,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:18,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:18,129 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:18,129 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,129 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:18,129 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,129 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,129 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,129 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:18,129 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:18,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:18,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:18,129 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:18,230 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:18,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:18,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:18,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:18,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:18,233 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:18,233 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:18,233 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:18,233 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:18,234 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,234 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:18,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844238232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:18,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844238232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,235 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,235 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:18,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:18,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:18,236 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,236 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:18,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:18,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:18,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:18,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:18,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:18,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:18,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:18,237 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:18,237 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:18,237 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:18,237 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:18,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:18,238 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:18,238 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,238 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:18,238 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:18,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,240 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:18,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:18,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:18,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:18,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:18,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:18,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:18,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79e82832[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:18,242 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:18,242 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,243 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:18,244 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,244 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:18,245 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,245 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:18,245 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:18,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:18,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:18,245 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:18,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:18,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:18,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:18,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:18,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:18,248 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:18,248 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:18,248 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:18,249 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:18,249 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:18,250 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,250 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,250 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:18,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:18,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:18,251 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:18,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,252 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:18,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,256 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,256 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:18,256 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:18,256 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,256 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:18,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:18,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:18,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:18,256 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,257 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,257 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,257 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,256 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:18,257 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:18,257 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:18,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:18,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:18,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:18,262 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:18,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844238261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,266 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:18,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844238264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,313 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041855374420f14b4903bf30a24666dd6de5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041855374420f14b4903bf30a24666dd6de5 2023-04-18 18:56:18,319 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=211, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/4003c9cc3cee44d9b18e213c94152fc4 2023-04-18 18:56:18,357 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:18,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:18,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:18,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:18,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:18,360 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:18,360 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:18,360 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:18,360 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:18,361 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,361 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,362 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:18,362 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:18,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:18,362 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,362 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:18,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:18,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:18,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:18,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:18,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:18,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:18,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:18,364 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:18,364 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:18,364 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:18,364 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:18,364 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:18,364 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:18,364 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:18,364 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:18,364 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,364 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:18,364 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:18,365 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,365 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:18,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:18,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:18,366 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,366 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:18,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:18,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:18,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:18,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@137a70cc[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:18,367 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:18,367 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,372 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,372 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,373 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:18,373 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:18,373 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:18,373 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:18,374 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:18,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:18,374 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:18,374 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:18,374 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:18,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:18,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:18,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:18,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:18,377 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:18,377 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:18,377 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:18,377 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:18,377 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:18,378 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,378 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,378 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:18,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:18,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:18,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,379 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:18,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:18,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:18,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:18,381 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:18,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,383 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,383 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:18,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:18,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:18,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:18,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:18,383 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:18,383 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:18,383 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:18,383 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,384 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,384 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:18,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:18,384 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:18,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:18,384 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,384 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,384 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,484 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:18,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:18,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:18,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:18,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:18,487 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:18,487 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:18,487 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:18,487 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:18,488 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,488 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,490 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:18,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:18,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:18,490 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,490 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,490 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:18,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:18,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:18,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:18,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:18,491 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:18,491 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:18,491 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:18,491 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:18,491 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:18,492 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:18,492 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:18,492 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:18,492 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:18,492 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:18,492 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:18,492 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,492 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:18,492 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:18,493 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,493 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:18,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:18,494 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,494 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:18,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:18,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:18,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:18,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:18,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fcaf0bc[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:18,496 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:18,496 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,497 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,497 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:18,497 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:18,497 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:18,498 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,498 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:18,499 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:18,499 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:18,499 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:18,499 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:18,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:18,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:18,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:18,502 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:18,502 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:18,502 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:18,502 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:18,502 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:18,504 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,505 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,505 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:18,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:18,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:18,505 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:18,507 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:18,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,509 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:18,509 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:18,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:18,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:18,509 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,509 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,509 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,509 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,509 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,509 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,509 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,509 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:18,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:18,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:18,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:18,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:18,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:18,510 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:18,510 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:18,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:18,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844238535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,538 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:18,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844238536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:18,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844238564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:18,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844238567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,610 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:18,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:18,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:18,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:18,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:18,613 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:18,613 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:18,614 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:18,614 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:18,614 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,614 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,615 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:18,615 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:18,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,616 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,616 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:18,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:18,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:18,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:18,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:18,618 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:18,618 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:18,618 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:18,618 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:18,618 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:18,618 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:18,618 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:18,618 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:18,618 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:18,618 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:18,618 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:18,618 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,619 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:18,619 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:18,620 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,620 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:18,621 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,621 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:18,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:18,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59272511[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:18,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:18,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,625 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,625 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:18,625 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:18,625 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,625 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:18,625 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:18,626 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:18,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:18,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:18,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:18,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:18,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:18,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:18,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:18,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:18,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:18,629 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:18,629 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:18,629 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:18,629 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:18,629 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:18,630 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,630 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,630 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:18,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:18,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:18,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:18,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:18,631 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:18,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:18,632 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:18,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:18,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,635 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,635 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,635 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,635 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:18,635 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:18,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:18,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:18,635 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,635 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,635 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:18,635 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,635 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:18,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:18,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:18,635 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:18,636 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:18,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:18,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:18,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:18,729 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/fcdfa72ef109489eb1c733748c86f3a1 2023-04-18 18:56:18,736 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:18,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:18,738 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/70804a300c8243768d488a85bfc653f1 2023-04-18 18:56:18,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:18,742 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/4003c9cc3cee44d9b18e213c94152fc4 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/4003c9cc3cee44d9b18e213c94152fc4 2023-04-18 18:56:18,745 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/4003c9cc3cee44d9b18e213c94152fc4, entries=400, sequenceid=211, filesize=59.4 K 2023-04-18 18:56:18,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:18,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:18,745 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:18,745 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:18,745 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:18,745 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:18,746 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,746 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,746 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/fcdfa72ef109489eb1c733748c86f3a1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fcdfa72ef109489eb1c733748c86f3a1 2023-04-18 18:56:18,749 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fcdfa72ef109489eb1c733748c86f3a1, entries=150, sequenceid=211, filesize=11.8 K 2023-04-18 18:56:18,750 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/70804a300c8243768d488a85bfc653f1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/70804a300c8243768d488a85bfc653f1 2023-04-18 18:56:18,753 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:18,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:18,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:18,753 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/70804a300c8243768d488a85bfc653f1, entries=150, sequenceid=211, filesize=11.8 K 2023-04-18 18:56:18,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,754 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:18,754 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:18,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:18,754 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for c48561c851d33e29da879cafb393a8fd in 856ms, sequenceid=211, compaction requested=true 2023-04-18 18:56:18,754 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:18,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:18,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:18,754 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:18,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:18,754 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:56:18,754 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:56:18,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:18,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:18,754 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:18,755 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:18,755 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:18,755 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:18,755 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:18,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:18,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:18,756 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:18,756 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:18,756 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 138850 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:56:18,756 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/A is initiating minor compaction (all files) 2023-04-18 18:56:18,756 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/A in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:18,756 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-18 18:56:18,756 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/89f97c0dc85c4c4195e82a1b9b141337, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/bb4b45fa96664dadaa72ffb577b9bd40, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/2fc3aee7c71149d9b02450907900d77d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/4003c9cc3cee44d9b18e213c94152fc4] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=135.6 K 2023-04-18 18:56:18,757 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:18,757 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:18,757 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:18,757 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:18,757 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:56:18,757 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 89f97c0dc85c4c4195e82a1b9b141337, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681844176061 2023-04-18 18:56:18,757 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/B is initiating minor compaction (all files) 2023-04-18 18:56:18,757 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/B in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:18,757 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:18,757 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b970553f344c4953b30a213313232a6c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/3302e82a60964e80a8ac241c16c32f2f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/4d124fd5e4ce413987828181719d8a05, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fcdfa72ef109489eb1c733748c86f3a1] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=47.3 K 2023-04-18 18:56:18,757 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting bb4b45fa96664dadaa72ffb577b9bd40, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1681844176130 2023-04-18 18:56:18,757 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:18,758 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b970553f344c4953b30a213313232a6c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681844176061 2023-04-18 18:56:18,758 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 2fc3aee7c71149d9b02450907900d77d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681844176749 2023-04-18 18:56:18,758 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 3302e82a60964e80a8ac241c16c32f2f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1681844176130 2023-04-18 18:56:18,758 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 4003c9cc3cee44d9b18e213c94152fc4, keycount=400, bloomtype=ROW, size=59.4 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1681844177889 2023-04-18 18:56:18,758 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 4d124fd5e4ce413987828181719d8a05, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681844176749 2023-04-18 18:56:18,764 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting fcdfa72ef109489eb1c733748c86f3a1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1681844177896 2023-04-18 18:56:18,778 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#B#compaction#586 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:18,785 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#A#compaction#584 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:19,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:19,045 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:19,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:19,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:19,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844239058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844239058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:19,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844239068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:19,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844239072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:19,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844239165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:19,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844239165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,198 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041862a98af4d39f4112b859540f39a7627c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041862a98af4d39f4112b859540f39a7627c 2023-04-18 18:56:19,204 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=247, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/9afd9b7966e44bb8bcf6832e9cc79aa8 2023-04-18 18:56:19,213 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/349845958e134ee68ba9fb0e364f386d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/349845958e134ee68ba9fb0e364f386d 2023-04-18 18:56:19,223 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c48561c851d33e29da879cafb393a8fd/B of c48561c851d33e29da879cafb393a8fd into 349845958e134ee68ba9fb0e364f386d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:19,223 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:19,223 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/B, priority=12, startTime=1681844178754; duration=0sec 2023-04-18 18:56:19,223 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:19,223 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:56:19,224 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/19ba411a86cc4edebb0e655a46de6479 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/19ba411a86cc4edebb0e655a46de6479 2023-04-18 18:56:19,225 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:56:19,225 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/C is initiating minor compaction (all files) 2023-04-18 18:56:19,225 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/C in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:19,225 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/f14ee3a998be4ca89f736e1fa6adde27, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/b2d9d644ae7e4b6a8820dc348d1c173a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/78a7ebbe6df749199cf84f04c9cccac3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/70804a300c8243768d488a85bfc653f1] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=47.3 K 2023-04-18 18:56:19,225 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting f14ee3a998be4ca89f736e1fa6adde27, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681844176061 2023-04-18 18:56:19,226 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b2d9d644ae7e4b6a8820dc348d1c173a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1681844176130 2023-04-18 18:56:19,226 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 78a7ebbe6df749199cf84f04c9cccac3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681844176749 2023-04-18 18:56:19,227 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 70804a300c8243768d488a85bfc653f1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1681844177896 2023-04-18 18:56:19,228 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/c3f596b1844e42d793fe4d808524caa5 2023-04-18 18:56:19,230 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c48561c851d33e29da879cafb393a8fd/A of c48561c851d33e29da879cafb393a8fd into 19ba411a86cc4edebb0e655a46de6479(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:19,230 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:19,230 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/A, priority=12, startTime=1681844178754; duration=0sec 2023-04-18 18:56:19,230 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:19,236 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#C#compaction#588 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:19,254 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/527cdd887eb4479e90d8243bda973530 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/527cdd887eb4479e90d8243bda973530 2023-04-18 18:56:19,258 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c48561c851d33e29da879cafb393a8fd/C of c48561c851d33e29da879cafb393a8fd into 527cdd887eb4479e90d8243bda973530(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:19,258 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:19,258 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/C, priority=12, startTime=1681844178755; duration=0sec 2023-04-18 18:56:19,258 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:19,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:19,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:19,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844239369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844239369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,649 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/84f6298b75d54f04a939f79f70894434 2023-04-18 18:56:19,653 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/9afd9b7966e44bb8bcf6832e9cc79aa8 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/9afd9b7966e44bb8bcf6832e9cc79aa8 2023-04-18 18:56:19,657 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/9afd9b7966e44bb8bcf6832e9cc79aa8, entries=150, sequenceid=247, filesize=25.3 K 2023-04-18 18:56:19,657 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/c3f596b1844e42d793fe4d808524caa5 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c3f596b1844e42d793fe4d808524caa5 2023-04-18 18:56:19,661 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c3f596b1844e42d793fe4d808524caa5, entries=150, sequenceid=247, filesize=11.8 K 2023-04-18 18:56:19,662 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/84f6298b75d54f04a939f79f70894434 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/84f6298b75d54f04a939f79f70894434 2023-04-18 18:56:19,666 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/84f6298b75d54f04a939f79f70894434, entries=150, sequenceid=247, filesize=11.8 K 2023-04-18 18:56:19,667 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for c48561c851d33e29da879cafb393a8fd in 911ms, sequenceid=247, compaction requested=false 2023-04-18 18:56:19,667 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:19,667 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:19,667 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:19,667 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:19,667 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,667 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:19,667 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:19,669 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,669 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:19,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:19,669 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,669 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:19,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:19,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:19,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:19,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:19,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:19,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61b63fbf[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:19,671 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:19,671 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,672 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,672 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:19,672 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:19,672 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:19,674 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:19,674 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:19,674 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:19,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:19,674 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-18 18:56:19,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:19,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:19,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:19,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:19,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:19,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:19,677 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,678 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:19,678 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:19,678 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:19,678 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:19,678 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:19,678 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:19,678 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:19,683 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,683 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,683 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:19,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:19,684 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:19,686 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,688 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:19,689 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:19,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:19,689 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:19,689 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:19,689 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:19,689 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:19,689 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,689 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,689 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,689 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:19,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:19,689 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:19,689 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:19,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:19,690 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418387888d0a69e49dfba2bc36f23bb8f17 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418387888d0a69e49dfba2bc36f23bb8f17 2023-04-18 18:56:19,696 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=262, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/9a78b8d44a9540dfa3e0f78aedf13bf2 2023-04-18 18:56:19,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:19,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844239707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:19,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844239708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,790 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:19,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:19,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:19,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:19,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:19,793 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:19,793 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:19,793 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:19,793 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:19,794 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,794 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:19,795 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:19,795 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:19,795 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,795 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:19,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:19,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:19,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:19,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:19,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:19,796 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:19,797 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:19,797 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:19,797 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:19,797 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:19,797 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:19,797 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:19,798 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:19,798 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:19,798 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:19,798 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:19,798 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,798 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:19,798 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:19,799 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,799 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:19,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:19,800 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,800 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:19,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:19,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:19,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:19,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:19,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:19,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@218adf55[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:19,801 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:19,801 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,803 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,803 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:19,803 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:19,803 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,803 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:19,803 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:19,805 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:19,805 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:19,805 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:19,805 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:19,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:19,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:19,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:19,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:19,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:19,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:19,809 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:19,809 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:19,809 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:19,809 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:19,809 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:19,811 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:19,811 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:19,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,811 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:19,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:19,812 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:19,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:19,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844239811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:19,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844239812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:19,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:19,814 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:19,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:19,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,817 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,817 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:19,817 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:19,817 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:19,817 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:19,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:19,817 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,817 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:19,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:19,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:19,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:19,817 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,817 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,817 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,817 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:19,818 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:19,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:19,918 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:19,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:19,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:19,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:19,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:19,921 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:19,921 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:19,921 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:19,921 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:19,922 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,922 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:19,924 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,924 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:19,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:19,924 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,924 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:19,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:19,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:19,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:19,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:19,926 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:19,926 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:19,926 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:19,926 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:19,926 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:19,926 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:19,926 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:19,926 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:19,926 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:19,926 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:19,926 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:19,926 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,926 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:19,926 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:19,928 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,928 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:19,928 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:19,928 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:19,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:19,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@492d94c8[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:19,930 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:19,930 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,931 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,931 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:19,931 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:19,931 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:19,931 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,931 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:19,932 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,932 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:19,932 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:19,932 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:19,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:19,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:19,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:19,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:19,935 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:19,935 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:19,935 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:19,935 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:19,935 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:19,937 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,937 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:19,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:19,938 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,939 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,941 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:19,941 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:19,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:19,941 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,941 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:19,941 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:19,941 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:19,941 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,941 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:19,941 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:19,941 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:19,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:19,944 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:19,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:19,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:19,944 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:19,944 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:19,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:20,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:20,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844240014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:20,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844240014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,044 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:20,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:20,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:20,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:20,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:20,047 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:20,047 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:20,047 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:20,047 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:20,048 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,048 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:20,049 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,049 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:20,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:20,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:20,050 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,050 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:20,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:20,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:20,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:20,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:20,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:20,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:20,051 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:20,051 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:20,051 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:20,051 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:20,051 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:20,051 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:20,051 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:20,051 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:20,051 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:20,051 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:20,051 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:20,051 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,051 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:20,051 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:20,053 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,053 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:20,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:20,054 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,054 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:20,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:20,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:20,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29423f51[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:20,055 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:20,055 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,057 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,057 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:20,057 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:20,057 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:20,057 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,057 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:20,058 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:20,059 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:20,059 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:20,059 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:20,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:20,061 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:20,061 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:20,061 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:20,061 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:20,061 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:20,062 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,062 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:20,062 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:20,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,064 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:20,065 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:20,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844240071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,076 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,076 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:20,076 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:20,076 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:20,076 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:20,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:20,076 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,076 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,076 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,076 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,077 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:20,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:20,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:20,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:20,077 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:20,077 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:20,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:20,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844240083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,105 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/d33d9be279f041319ac29246a5bfe1ae 2023-04-18 18:56:20,113 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/6745637e850a43f1ae0463d84078abe1 2023-04-18 18:56:20,117 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/9a78b8d44a9540dfa3e0f78aedf13bf2 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/9a78b8d44a9540dfa3e0f78aedf13bf2 2023-04-18 18:56:20,120 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/9a78b8d44a9540dfa3e0f78aedf13bf2, entries=150, sequenceid=262, filesize=25.4 K 2023-04-18 18:56:20,121 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/d33d9be279f041319ac29246a5bfe1ae as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/d33d9be279f041319ac29246a5bfe1ae 2023-04-18 18:56:20,124 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/d33d9be279f041319ac29246a5bfe1ae, entries=100, sequenceid=262, filesize=9.5 K 2023-04-18 18:56:20,125 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/6745637e850a43f1ae0463d84078abe1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/6745637e850a43f1ae0463d84078abe1 2023-04-18 18:56:20,127 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/6745637e850a43f1ae0463d84078abe1, entries=100, sequenceid=262, filesize=9.5 K 2023-04-18 18:56:20,128 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c48561c851d33e29da879cafb393a8fd in 454ms, sequenceid=262, compaction requested=true 2023-04-18 18:56:20,128 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:20,128 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:20,128 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:20,128 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:20,128 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:20,128 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:20,129 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78228 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:20,129 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34208 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:20,129 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/A is initiating minor compaction (all files) 2023-04-18 18:56:20,129 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/B is initiating minor compaction (all files) 2023-04-18 18:56:20,129 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/A in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:20,130 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/B in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:20,130 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/19ba411a86cc4edebb0e655a46de6479, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/9afd9b7966e44bb8bcf6832e9cc79aa8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/9a78b8d44a9540dfa3e0f78aedf13bf2] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=76.4 K 2023-04-18 18:56:20,130 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/349845958e134ee68ba9fb0e364f386d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c3f596b1844e42d793fe4d808524caa5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/d33d9be279f041319ac29246a5bfe1ae] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=33.4 K 2023-04-18 18:56:20,131 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 19ba411a86cc4edebb0e655a46de6479, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1681844177896 2023-04-18 18:56:20,131 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 349845958e134ee68ba9fb0e364f386d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1681844177896 2023-04-18 18:56:20,131 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 9afd9b7966e44bb8bcf6832e9cc79aa8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1681844177949 2023-04-18 18:56:20,131 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting c3f596b1844e42d793fe4d808524caa5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1681844177949 2023-04-18 18:56:20,132 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 9a78b8d44a9540dfa3e0f78aedf13bf2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1681844179045 2023-04-18 18:56:20,132 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting d33d9be279f041319ac29246a5bfe1ae, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1681844179045 2023-04-18 18:56:20,138 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#B#compaction#594 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:20,145 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#A#compaction#593 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:20,177 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:20,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:20,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:20,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:20,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:20,180 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:20,180 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:20,180 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:20,180 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:20,181 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,181 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:20,182 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:20,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:20,182 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:20,183 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,183 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:20,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:20,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:20,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:20,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:20,184 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:20,184 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:20,184 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 18:56:20,184 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:20,184 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:20,185 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:20,185 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:20,185 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:20,185 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:20,202 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418f129450d5b574acbacc6ee2707dbc7f9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f129450d5b574acbacc6ee2707dbc7f9 2023-04-18 18:56:20,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:20,238 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:20,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:20,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844240261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:20,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43886 deadline: 1681844240317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:20,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43916 deadline: 1681844240317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:20,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844240364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,558 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/850146392e4b4b29895c506e5bd75fd4 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/850146392e4b4b29895c506e5bd75fd4 2023-04-18 18:56:20,562 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c48561c851d33e29da879cafb393a8fd/B of c48561c851d33e29da879cafb393a8fd into 850146392e4b4b29895c506e5bd75fd4(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:20,562 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:20,562 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/B, priority=13, startTime=1681844180128; duration=0sec 2023-04-18 18:56:20,562 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:20,562 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:20,563 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34208 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:20,563 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/C is initiating minor compaction (all files) 2023-04-18 18:56:20,563 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/C in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:20,563 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/527cdd887eb4479e90d8243bda973530, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/84f6298b75d54f04a939f79f70894434, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/6745637e850a43f1ae0463d84078abe1] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=33.4 K 2023-04-18 18:56:20,563 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 527cdd887eb4479e90d8243bda973530, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1681844177896 2023-04-18 18:56:20,564 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 84f6298b75d54f04a939f79f70894434, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1681844177949 2023-04-18 18:56:20,564 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 6745637e850a43f1ae0463d84078abe1, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1681844179045 2023-04-18 18:56:20,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:20,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844240568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,575 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#C#compaction#596 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:20,590 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/2695ce180caa4e68a39d3a6b40e25709 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/2695ce180caa4e68a39d3a6b40e25709 2023-04-18 18:56:20,591 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/d7ed443a5344422c8ebe9a9515cb81c4 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/d7ed443a5344422c8ebe9a9515cb81c4 2023-04-18 18:56:20,595 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c48561c851d33e29da879cafb393a8fd/C of c48561c851d33e29da879cafb393a8fd into 2695ce180caa4e68a39d3a6b40e25709(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:20,596 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:20,596 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/C, priority=13, startTime=1681844180128; duration=0sec 2023-04-18 18:56:20,596 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:20,596 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c48561c851d33e29da879cafb393a8fd/A of c48561c851d33e29da879cafb393a8fd into d7ed443a5344422c8ebe9a9515cb81c4(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:20,597 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:20,597 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/A, priority=13, startTime=1681844180128; duration=0sec 2023-04-18 18:56:20,597 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:20,607 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=282, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/a2ee948d72e743269fef7f009a89cadf 2023-04-18 18:56:20,616 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/c89b87f3f23d4be4a2fa9cee24de5681 2023-04-18 18:56:20,625 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/e5b02b316f84465486c1a4d41832b7ce 2023-04-18 18:56:20,628 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/a2ee948d72e743269fef7f009a89cadf as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/a2ee948d72e743269fef7f009a89cadf 2023-04-18 18:56:20,632 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/a2ee948d72e743269fef7f009a89cadf, entries=150, sequenceid=282, filesize=25.4 K 2023-04-18 18:56:20,632 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/c89b87f3f23d4be4a2fa9cee24de5681 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c89b87f3f23d4be4a2fa9cee24de5681 2023-04-18 18:56:20,636 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c89b87f3f23d4be4a2fa9cee24de5681, entries=150, sequenceid=282, filesize=11.9 K 2023-04-18 18:56:20,637 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/e5b02b316f84465486c1a4d41832b7ce as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/e5b02b316f84465486c1a4d41832b7ce 2023-04-18 18:56:20,640 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/e5b02b316f84465486c1a4d41832b7ce, entries=150, sequenceid=282, filesize=11.9 K 2023-04-18 18:56:20,640 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for c48561c851d33e29da879cafb393a8fd in 456ms, sequenceid=282, compaction requested=false 2023-04-18 18:56:20,640 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:20,640 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:20,640 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:20,641 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:20,641 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,641 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:20,641 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:20,642 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,642 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:20,643 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,643 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:20,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:20,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:20,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:20,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:20,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79d9b601[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:20,644 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:20,644 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,645 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,646 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:20,646 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:20,646 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,646 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:20,646 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:20,648 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,649 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:20,649 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:20,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:20,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:20,649 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:20,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:20,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:20,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:20,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:20,652 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:20,652 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:20,652 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:20,652 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:20,652 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:20,653 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,653 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,653 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:20,654 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:20,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:20,655 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:20,658 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,658 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:20,658 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:20,658 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,658 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,658 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:20,659 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:20,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:20,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:20,659 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:20,659 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:20,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:20,659 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:20,659 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:20,660 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:20,760 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:20,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:20,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:20,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:20,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:20,763 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:20,763 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:20,763 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:20,763 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:20,764 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,764 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:20,765 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:20,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:20,766 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,766 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,766 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:20,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:20,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:20,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:20,767 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:20,767 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:20,767 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:20,767 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:20,767 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:20,767 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:20,767 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:20,768 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-18 18:56:20,768 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:20,768 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:20,768 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:20,768 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:20,768 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:20,768 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:20,780 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418a29d62a728964a629db081a72fc0bfd0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a29d62a728964a629db081a72fc0bfd0 2023-04-18 18:56:20,786 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=297, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/99088948e48a46f3a41a48d91cbcc21d 2023-04-18 18:56:20,794 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/5016fed21fb64ded8b0356007c813113 2023-04-18 18:56:20,818 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/8ebd59f4618c4fedb0379bc644f075cc 2023-04-18 18:56:20,822 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/99088948e48a46f3a41a48d91cbcc21d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/99088948e48a46f3a41a48d91cbcc21d 2023-04-18 18:56:20,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:20,824 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:20,827 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/99088948e48a46f3a41a48d91cbcc21d, entries=150, sequenceid=297, filesize=25.4 K 2023-04-18 18:56:20,828 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/5016fed21fb64ded8b0356007c813113 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/5016fed21fb64ded8b0356007c813113 2023-04-18 18:56:20,831 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/5016fed21fb64ded8b0356007c813113, entries=150, sequenceid=297, filesize=11.9 K 2023-04-18 18:56:20,832 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/8ebd59f4618c4fedb0379bc644f075cc as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/8ebd59f4618c4fedb0379bc644f075cc 2023-04-18 18:56:20,836 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/8ebd59f4618c4fedb0379bc644f075cc, entries=150, sequenceid=297, filesize=11.9 K 2023-04-18 18:56:20,836 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=40.25 KB/41220 for c48561c851d33e29da879cafb393a8fd in 68ms, sequenceid=297, compaction requested=true 2023-04-18 18:56:20,837 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:20,837 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:20,837 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:20,837 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:20,837 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,837 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:20,837 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:20,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:20,839 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-18 18:56:20,840 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:20,840 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:20,840 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:20,840 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:20,840 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:20,840 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:20,841 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,841 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:20,842 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,842 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:20,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:20,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e5eae18[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:20,844 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:20,844 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,845 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,846 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:20,846 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:20,846 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:20,846 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,846 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:20,847 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:20,847 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:20,847 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:20,848 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:20,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:20,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:20,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:20,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:20,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:20,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:20,850 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:20,851 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:20,851 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:20,851 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:20,851 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:20,852 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,852 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:20,852 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:20,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,853 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:20,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,855 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,857 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418e76292350b7e457a9516f80593cd78bf to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e76292350b7e457a9516f80593cd78bf 2023-04-18 18:56:20,858 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,858 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:20,858 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:20,858 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,858 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,858 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,858 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,858 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:20,858 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:20,858 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:20,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:20,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:20,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:20,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:20,859 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:20,859 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:20,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:20,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:20,862 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=308, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/04de38b715764571a68e3f36f6bb80d6 2023-04-18 18:56:20,871 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=308 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/97daa835736f4183bb7e293500726ee1 2023-04-18 18:56:20,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844180862, max timestamp=1681844180875Num uniques 859; 2023-04-18 18:56:20,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:56:20,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844180862, max timestamp=1681844180875Num uniques 859; 2023-04-18 18:56:20,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:56:20,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844180862, max timestamp=1681844180875Num uniques 859; 2023-04-18 18:56:20,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:56:20,880 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=308 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/4e3867d26c324716a522732929ed47b1 2023-04-18 18:56:20,884 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/04de38b715764571a68e3f36f6bb80d6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/04de38b715764571a68e3f36f6bb80d6 2023-04-18 18:56:20,887 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/04de38b715764571a68e3f36f6bb80d6, entries=150, sequenceid=308, filesize=25.4 K 2023-04-18 18:56:20,888 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/97daa835736f4183bb7e293500726ee1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/97daa835736f4183bb7e293500726ee1 2023-04-18 18:56:20,895 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/97daa835736f4183bb7e293500726ee1, entries=150, sequenceid=308, filesize=11.9 K 2023-04-18 18:56:20,895 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/4e3867d26c324716a522732929ed47b1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4e3867d26c324716a522732929ed47b1 2023-04-18 18:56:20,899 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4e3867d26c324716a522732929ed47b1, entries=150, sequenceid=308, filesize=11.9 K 2023-04-18 18:56:20,900 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=107.34 KB/109920 for c48561c851d33e29da879cafb393a8fd in 61ms, sequenceid=308, compaction requested=true 2023-04-18 18:56:20,900 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:20,900 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:20,900 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:56:20,900 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:20,900 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:56:20,901 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:20,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:20,901 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=120.76 KB heapSize=345.14 KB 2023-04-18 18:56:20,902 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:56:20,902 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/A is initiating minor compaction (all files) 2023-04-18 18:56:20,902 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/A in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:20,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:20,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:20,902 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49190 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:56:20,902 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/B is initiating minor compaction (all files) 2023-04-18 18:56:20,902 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/B in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:20,902 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/850146392e4b4b29895c506e5bd75fd4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c89b87f3f23d4be4a2fa9cee24de5681, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/5016fed21fb64ded8b0356007c813113, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/97daa835736f4183bb7e293500726ee1] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=48.0 K 2023-04-18 18:56:20,902 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/d7ed443a5344422c8ebe9a9515cb81c4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/a2ee948d72e743269fef7f009a89cadf, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/99088948e48a46f3a41a48d91cbcc21d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/04de38b715764571a68e3f36f6bb80d6] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=102.2 K 2023-04-18 18:56:20,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:20,903 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:20,903 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:20,903 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:20,903 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 850146392e4b4b29895c506e5bd75fd4, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1681844177953 2023-04-18 18:56:20,903 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting c89b87f3f23d4be4a2fa9cee24de5681, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1681844179697 2023-04-18 18:56:20,903 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting d7ed443a5344422c8ebe9a9515cb81c4, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1681844177953 2023-04-18 18:56:20,904 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 5016fed21fb64ded8b0356007c813113, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1681844180246 2023-04-18 18:56:20,904 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting a2ee948d72e743269fef7f009a89cadf, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1681844179697 2023-04-18 18:56:20,904 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 97daa835736f4183bb7e293500726ee1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681844180831 2023-04-18 18:56:20,905 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 99088948e48a46f3a41a48d91cbcc21d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1681844180246 2023-04-18 18:56:20,905 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 04de38b715764571a68e3f36f6bb80d6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681844180831 2023-04-18 18:56:20,917 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#B#compaction#607 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:20,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:20,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844240916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:20,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844240918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:20,923 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#A#compaction#606 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:20,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844240919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,937 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/3c438fad9a404731a972ba8893ca5eb8 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/3c438fad9a404731a972ba8893ca5eb8 2023-04-18 18:56:20,942 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c48561c851d33e29da879cafb393a8fd/B of c48561c851d33e29da879cafb393a8fd into 3c438fad9a404731a972ba8893ca5eb8(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:20,942 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:20,942 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/B, priority=12, startTime=1681844180900; duration=0sec 2023-04-18 18:56:20,942 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:20,942 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:56:20,943 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49190 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:56:20,943 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/C is initiating minor compaction (all files) 2023-04-18 18:56:20,943 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/C in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:20,944 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/2695ce180caa4e68a39d3a6b40e25709, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/e5b02b316f84465486c1a4d41832b7ce, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/8ebd59f4618c4fedb0379bc644f075cc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4e3867d26c324716a522732929ed47b1] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=48.0 K 2023-04-18 18:56:20,944 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 2695ce180caa4e68a39d3a6b40e25709, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1681844177953 2023-04-18 18:56:20,944 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting e5b02b316f84465486c1a4d41832b7ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1681844179697 2023-04-18 18:56:20,945 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 8ebd59f4618c4fedb0379bc644f075cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1681844180246 2023-04-18 18:56:20,945 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 4e3867d26c324716a522732929ed47b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681844180831 2023-04-18 18:56:20,957 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#C#compaction#608 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:20,959 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:20,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:20,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:20,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:20,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:20,963 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:20,963 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:20,963 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:20,963 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:20,964 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,964 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:20,966 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:20,966 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:20,966 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/2cccb0f4958647be918e040091173531 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/2cccb0f4958647be918e040091173531 2023-04-18 18:56:20,967 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,967 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:20,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:20,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:20,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:20,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:20,968 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:20,968 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:20,968 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:20,969 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:20,969 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:20,969 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:20,969 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:20,969 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:20,969 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:20,969 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:20,969 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:20,969 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,969 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:20,969 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:20,971 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,971 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:20,971 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c48561c851d33e29da879cafb393a8fd/C of c48561c851d33e29da879cafb393a8fd into 2cccb0f4958647be918e040091173531(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:20,971 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:20,971 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/C, priority=12, startTime=1681844180901; duration=0sec 2023-04-18 18:56:20,971 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:20,971 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,972 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:20,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:20,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bf0f0cf[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:20,973 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:20,973 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,974 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,974 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:20,975 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:20,975 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:20,975 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,975 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:20,977 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:20,977 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:20,977 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,977 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:20,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:20,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:20,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:20,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:20,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:20,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:20,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:20,980 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:20,980 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:20,980 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:20,980 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:20,980 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:20,982 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:20,982 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,982 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:20,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:20,982 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:20,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:20,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,984 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,989 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,989 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:20,989 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:20,989 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:20,989 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,989 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:20,989 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:20,989 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:20,989 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:20,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:20,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:20,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:20,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:20,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:20,989 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:20,990 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:20,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:20,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:20,990 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:21,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:21,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844241020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:21,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844241022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:21,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844241024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,090 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:21,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:21,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:21,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:21,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:21,093 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:21,094 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:21,094 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:21,094 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:21,094 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,094 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,096 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:21,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:21,096 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,097 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,097 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:21,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:21,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:21,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:21,098 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:21,098 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:21,098 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:21,098 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:21,098 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:21,098 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:21,098 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:21,098 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:21,098 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:21,098 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:21,099 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:21,099 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,099 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:21,099 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:21,100 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,100 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:21,100 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,100 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:21,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:21,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1370fa57[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:21,102 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:21,102 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,103 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,103 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:21,103 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:21,103 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:21,103 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,103 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:21,104 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:21,104 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:21,105 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:21,105 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:21,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:21,107 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:21,107 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:21,107 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:21,107 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:21,107 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:21,110 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,110 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:21,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,110 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:21,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:21,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,111 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:21,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,112 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,115 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:21,115 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:21,115 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,115 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,115 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,115 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,115 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,115 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,115 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:21,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:21,115 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:21,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:21,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:21,116 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:21,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:21,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:21,116 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:21,216 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:21,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:21,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:21,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:21,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:21,219 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:21,219 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:21,219 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:21,219 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:21,220 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,220 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,221 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,221 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:21,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:21,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:21,221 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,221 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:21,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:21,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:21,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:21,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:21,223 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:21,223 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:21,223 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:21,223 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:21,223 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:21,223 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:21,224 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:21,224 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:21,224 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:21,225 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,225 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:21,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:21,226 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:21,226 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:21,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:21,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844241223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:21,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:21,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:21,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844241226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,228 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:21,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@272d2f86[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:21,228 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:21,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844241228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,230 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,230 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:21,230 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:21,230 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:21,230 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,230 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:21,232 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:21,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:21,232 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:21,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:21,234 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:21,234 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:21,234 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:21,234 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:21,234 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:21,237 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,237 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:21,237 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:21,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:21,237 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,240 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,243 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:21,243 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,243 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:21,243 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:21,243 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:21,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:21,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:21,243 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,244 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,244 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,244 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,244 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,244 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:21,244 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:21,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:21,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:21,327 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418596f95aa4aa34a4381d1345d676e6cd9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418596f95aa4aa34a4381d1345d676e6cd9 2023-04-18 18:56:21,333 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=343, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/7aa2256b090f453e9284ef2439654a01 2023-04-18 18:56:21,344 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:21,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:21,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:21,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:21,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:21,352 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:21,352 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:21,352 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:21,352 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:21,353 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,353 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,356 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:21,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:21,356 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,357 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,357 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:21,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:21,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:21,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:21,359 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:21,359 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/4f5139d197f242289c581cc65378a915 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/4f5139d197f242289c581cc65378a915 2023-04-18 18:56:21,359 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:21,359 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:21,359 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:21,359 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:21,359 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:21,359 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:21,360 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:21,359 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:21,361 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:21,361 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:21,361 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,361 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:21,361 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:21,362 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,362 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:21,363 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c48561c851d33e29da879cafb393a8fd/A of c48561c851d33e29da879cafb393a8fd into 4f5139d197f242289c581cc65378a915(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:21,363 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:21,363 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/A, priority=12, startTime=1681844180900; duration=0sec 2023-04-18 18:56:21,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,363 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:21,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:21,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:21,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cf86b15[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:21,365 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:21,365 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,366 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,366 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:21,366 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:21,366 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:21,366 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,367 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:21,368 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:21,368 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,368 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:21,368 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:21,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:21,370 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:21,371 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:21,371 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:21,371 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:21,371 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:21,373 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,373 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,373 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:21,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:21,373 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:21,378 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:21,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:21,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:21,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:21,380 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:21,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:21,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:21,381 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:21,381 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:21,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:21,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:21,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:21,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:21,481 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:21,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:21,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:21,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:21,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:21,484 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:21,484 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:21,484 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:21,484 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:21,485 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,485 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,486 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:21,486 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:21,486 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,487 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:21,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:21,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:21,488 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:21,488 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:21,488 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:21,488 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:21,489 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:21,489 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:21,489 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,489 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:21,489 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:21,493 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,493 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:21,493 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,493 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:21,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:21,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70228ecd[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:21,495 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:21,495 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,497 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,497 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:21,497 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:21,498 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:21,498 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,498 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:21,499 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:21,499 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:21,499 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:21,500 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:21,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:21,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:21,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:21,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:21,503 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:21,503 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:21,504 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:21,504 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:21,504 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:21,505 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,505 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:21,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,505 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:21,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:21,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:21,506 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,507 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,511 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,511 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:21,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:21,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:21,511 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,511 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,511 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,511 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,511 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,511 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:21,511 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:21,511 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:21,511 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,511 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:21,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:21,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:21,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:21,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:21,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:21,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844241528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:21,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844241529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:21,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844241532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,612 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:21,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:21,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:21,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:21,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:21,615 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:21,615 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:21,615 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:21,615 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:21,616 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,616 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,617 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,617 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:21,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:21,617 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,617 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:21,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:21,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:21,618 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:21,618 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:21,618 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:21,619 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:21,619 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:21,619 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:21,619 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:21,619 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:21,620 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:21,620 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,621 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:21,621 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:21,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:21,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@641460dc[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:21,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:21,623 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,623 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,624 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,624 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:21,624 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:21,624 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:21,624 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:21,626 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:21,626 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:21,627 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:21,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:21,629 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:21,629 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:21,629 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:21,629 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:21,629 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:21,630 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,630 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:21,630 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:21,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:21,631 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,632 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:21,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,634 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:21,634 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:21,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:21,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:21,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:21,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:21,635 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:21,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:21,634 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,635 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,635 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,635 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,635 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,635 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,635 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,635 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:21,635 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:21,736 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:21,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:21,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:21,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:21,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:21,740 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:21,740 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:21,740 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:21,740 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:21,740 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,741 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,743 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:21,743 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:21,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:21,743 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,743 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:21,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:21,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:21,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:21,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:21,746 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:21,746 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:21,746 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:21,747 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:21,747 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:21,747 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:21,747 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:21,751 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,751 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:21,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:21,752 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,752 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:21,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:21,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:21,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,756 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/a187514ec0714421845a383f566396da 2023-04-18 18:56:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:21,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:21,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a0b32c7[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:21,757 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:21,757 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,758 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,758 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:21,758 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:21,758 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:21,758 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,758 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:21,759 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,759 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:21,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:21,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:21,759 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:21,759 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:21,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:21,766 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:21,766 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:21,766 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:21,766 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:21,766 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:21,768 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,768 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:21,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,768 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:21,769 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:21,770 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/adf1516194914331acf01e7b77f867b8 2023-04-18 18:56:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:21,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,772 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,775 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/7aa2256b090f453e9284ef2439654a01 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/7aa2256b090f453e9284ef2439654a01 2023-04-18 18:56:21,775 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:21,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:21,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:21,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:21,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:21,776 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:21,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:21,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,776 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:21,776 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:21,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:21,779 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/7aa2256b090f453e9284ef2439654a01, entries=250, sequenceid=343, filesize=39.2 K 2023-04-18 18:56:21,779 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/a187514ec0714421845a383f566396da as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/a187514ec0714421845a383f566396da 2023-04-18 18:56:21,783 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/a187514ec0714421845a383f566396da, entries=150, sequenceid=343, filesize=11.9 K 2023-04-18 18:56:21,784 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/adf1516194914331acf01e7b77f867b8 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/adf1516194914331acf01e7b77f867b8 2023-04-18 18:56:21,788 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/adf1516194914331acf01e7b77f867b8, entries=150, sequenceid=343, filesize=11.9 K 2023-04-18 18:56:21,788 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~345.09 KB/353376, currentSize=60.38 KB/61830 for c48561c851d33e29da879cafb393a8fd in 887ms, sequenceid=343, compaction requested=false 2023-04-18 18:56:21,788 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:21,876 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:21,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:21,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:21,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:21,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:21,880 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:21,880 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:21,880 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:21,880 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:21,880 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,880 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,881 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:21,881 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:21,882 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:21,882 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:21,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:21,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:21,883 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:21,883 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:21,883 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:21,884 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:21,884 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:21,884 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-18 18:56:21,884 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:21,884 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:21,884 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:21,884 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:21,885 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:21,885 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:22,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:22,035 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:22,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:22,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844242057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:22,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:22,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844242060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844242060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:22,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844242091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,095 DEBUG [Thread-3979] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:56:22,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:22,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844242096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,100 DEBUG [Thread-3973] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:56:22,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:22,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844242161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:22,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844242162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:22,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844242163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,305 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418df1b9554f8c844cba5fe0f93e4682dc4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418df1b9554f8c844cba5fe0f93e4682dc4 2023-04-18 18:56:22,310 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=358, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/bef46d0ed3774f0bb0b799ce8907a3dc 2023-04-18 18:56:22,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:22,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844242364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:22,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844242366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:22,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844242366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:22,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844242667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:22,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844242669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:22,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844242670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,718 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/256116a4659d4790bc566ba2c8e6b563 2023-04-18 18:56:22,727 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/c1b8c3370ad7424896b725be9bc1d275 2023-04-18 18:56:22,730 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/bef46d0ed3774f0bb0b799ce8907a3dc as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/bef46d0ed3774f0bb0b799ce8907a3dc 2023-04-18 18:56:22,733 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/bef46d0ed3774f0bb0b799ce8907a3dc, entries=100, sequenceid=358, filesize=18.6 K 2023-04-18 18:56:22,733 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/256116a4659d4790bc566ba2c8e6b563 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/256116a4659d4790bc566ba2c8e6b563 2023-04-18 18:56:22,736 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/256116a4659d4790bc566ba2c8e6b563, entries=100, sequenceid=358, filesize=9.5 K 2023-04-18 18:56:22,737 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/c1b8c3370ad7424896b725be9bc1d275 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/c1b8c3370ad7424896b725be9bc1d275 2023-04-18 18:56:22,740 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/c1b8c3370ad7424896b725be9bc1d275, entries=100, sequenceid=358, filesize=9.5 K 2023-04-18 18:56:22,742 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c48561c851d33e29da879cafb393a8fd in 858ms, sequenceid=358, compaction requested=true 2023-04-18 18:56:22,742 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:22,742 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:22,742 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:22,742 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:22,742 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,742 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:22,742 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:22,744 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:22,747 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:22,747 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:22,747 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:22,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:22,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35b44450[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:22,749 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:22,749 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:22,754 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:22,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:22,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:22,754 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:22,754 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:22,754 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:22,755 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,755 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:22,755 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:22,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:22,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:22,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:22,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:22,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:22,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:22,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:22,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:22,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:22,759 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:22,759 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:22,759 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:22,759 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:22,759 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:22,761 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:22,761 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:22,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:22,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:22,761 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:22,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:22,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:22,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:22,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:22,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:22,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:22,762 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:22,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:22,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,763 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,767 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,767 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:22,767 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:22,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:22,768 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:22,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:22,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:22,768 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:22,767 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:22,768 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:22,768 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:22,768 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:22,768 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:22,768 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:22,768 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:22,868 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:22,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:22,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:22,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:22,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:22,872 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:22,872 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:22,872 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:22,872 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:22,872 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:22,872 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:22,873 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,873 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:22,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:22,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:22,874 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:22,874 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:22,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:22,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:22,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:22,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:22,875 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:22,875 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:22,875 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:22,875 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:22,875 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:22,875 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:22,875 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:22,875 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 18:56:22,875 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:22,876 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:22,876 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:22,876 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:22,876 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:22,876 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:22,890 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304189073d42d6ab948358e636cfffc98bd75 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304189073d42d6ab948358e636cfffc98bd75 2023-04-18 18:56:23,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:23,174 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:23,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:23,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844243182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:23,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844243182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:23,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844243183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:23,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844243286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:23,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844243286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,289 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:23,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844243287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,314 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=378, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/70f1a5f4472f42d2b9d0d74af634497d 2023-04-18 18:56:23,323 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/c720c7c773d741159ac127fdb022d737 2023-04-18 18:56:23,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:23,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844243490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:23,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844243490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:23,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844243490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,732 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/02b687b02f5c4c23b60c1f57178c20a0 2023-04-18 18:56:23,736 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/70f1a5f4472f42d2b9d0d74af634497d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/70f1a5f4472f42d2b9d0d74af634497d 2023-04-18 18:56:23,739 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/70f1a5f4472f42d2b9d0d74af634497d, entries=150, sequenceid=378, filesize=25.4 K 2023-04-18 18:56:23,740 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/c720c7c773d741159ac127fdb022d737 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c720c7c773d741159ac127fdb022d737 2023-04-18 18:56:23,743 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c720c7c773d741159ac127fdb022d737, entries=150, sequenceid=378, filesize=11.9 K 2023-04-18 18:56:23,743 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/02b687b02f5c4c23b60c1f57178c20a0 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/02b687b02f5c4c23b60c1f57178c20a0 2023-04-18 18:56:23,746 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/02b687b02f5c4c23b60c1f57178c20a0, entries=150, sequenceid=378, filesize=11.9 K 2023-04-18 18:56:23,747 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for c48561c851d33e29da879cafb393a8fd in 872ms, sequenceid=378, compaction requested=true 2023-04-18 18:56:23,747 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:23,747 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:23,747 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:23,747 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:23,747 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,747 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:23,747 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:23,749 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,749 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:23,749 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:23,749 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:23,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:23,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:23,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:23,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5184e0cd[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:23,751 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:23,751 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:23,752 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:23,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:23,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:23,752 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:23,752 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:23,752 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:23,752 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,752 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:23,754 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:23,754 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:23,754 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,754 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:23,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:23,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:23,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:23,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:23,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:23,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:23,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:23,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:23,757 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:23,757 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:23,757 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:23,757 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:23,757 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:23,759 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:23,759 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:23,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:23,759 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:23,759 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:23,760 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:23,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:23,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,762 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,763 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:23,763 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:23,763 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:23,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:23,763 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:23,763 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:23,763 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:23,763 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:23,763 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:23,763 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:23,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:23,763 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:23,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:23,763 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:23,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:23,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:23,796 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 18:56:23,796 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:23,796 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:23,796 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:23,796 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:23,796 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:23,796 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:23,809 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418044149bd08f04fc49211debc122ccb62 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418044149bd08f04fc49211debc122ccb62 2023-04-18 18:56:23,814 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=392, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/1eefdae08ee146228dfd333297324f29 2023-04-18 18:56:23,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:23,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844243815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:23,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844243818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:23,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844243818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,824 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/b6fb5123883844a09e308ac6506c27b4 2023-04-18 18:56:23,835 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/c42d6373e1db44d28ee87ad35cd14ce6 2023-04-18 18:56:23,839 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/1eefdae08ee146228dfd333297324f29 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/1eefdae08ee146228dfd333297324f29 2023-04-18 18:56:23,842 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/1eefdae08ee146228dfd333297324f29, entries=200, sequenceid=392, filesize=32.3 K 2023-04-18 18:56:23,843 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/b6fb5123883844a09e308ac6506c27b4 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b6fb5123883844a09e308ac6506c27b4 2023-04-18 18:56:23,846 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b6fb5123883844a09e308ac6506c27b4, entries=150, sequenceid=392, filesize=11.9 K 2023-04-18 18:56:23,847 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/c42d6373e1db44d28ee87ad35cd14ce6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/c42d6373e1db44d28ee87ad35cd14ce6 2023-04-18 18:56:23,850 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/c42d6373e1db44d28ee87ad35cd14ce6, entries=150, sequenceid=392, filesize=11.9 K 2023-04-18 18:56:23,850 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c48561c851d33e29da879cafb393a8fd in 54ms, sequenceid=392, compaction requested=true 2023-04-18 18:56:23,850 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:23,851 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:23,851 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:56:23,851 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:56:23,851 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:23,851 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:23,853 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 144988 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:56:23,853 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/A is initiating minor compaction (all files) 2023-04-18 18:56:23,853 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/A in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:23,853 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/4f5139d197f242289c581cc65378a915, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/7aa2256b090f453e9284ef2439654a01, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/bef46d0ed3774f0bb0b799ce8907a3dc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/70f1a5f4472f42d2b9d0d74af634497d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/1eefdae08ee146228dfd333297324f29] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=141.6 K 2023-04-18 18:56:23,853 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59168 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:56:23,853 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/B is initiating minor compaction (all files) 2023-04-18 18:56:23,853 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/B in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:23,854 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/3c438fad9a404731a972ba8893ca5eb8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/a187514ec0714421845a383f566396da, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/256116a4659d4790bc566ba2c8e6b563, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c720c7c773d741159ac127fdb022d737, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b6fb5123883844a09e308ac6506c27b4] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=57.8 K 2023-04-18 18:56:23,854 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 4f5139d197f242289c581cc65378a915, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681844180831 2023-04-18 18:56:23,854 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 7aa2256b090f453e9284ef2439654a01, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1681844180883 2023-04-18 18:56:23,855 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 3c438fad9a404731a972ba8893ca5eb8, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681844180831 2023-04-18 18:56:23,855 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting bef46d0ed3774f0bb0b799ce8907a3dc, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681844180917 2023-04-18 18:56:23,855 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting a187514ec0714421845a383f566396da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1681844180883 2023-04-18 18:56:23,855 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 70f1a5f4472f42d2b9d0d74af634497d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1681844182053 2023-04-18 18:56:23,855 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 256116a4659d4790bc566ba2c8e6b563, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681844180917 2023-04-18 18:56:23,856 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 1eefdae08ee146228dfd333297324f29, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1681844183181 2023-04-18 18:56:23,856 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting c720c7c773d741159ac127fdb022d737, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1681844182053 2023-04-18 18:56:23,857 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b6fb5123883844a09e308ac6506c27b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1681844183181 2023-04-18 18:56:23,863 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:23,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:23,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:23,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:23,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:23,867 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:23,867 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:23,867 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:23,867 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:23,868 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:23,868 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:23,869 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:23,869 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:23,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:23,870 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:23,870 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#B#compaction#621 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:23,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:23,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:23,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:23,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:23,870 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:23,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:23,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:23,872 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:23,872 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:23,872 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:23,878 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#A#compaction#620 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:23,879 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:23,879 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:23,879 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:23,879 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:23,879 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-18 18:56:23,879 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:23,879 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:23,879 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:23,879 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:23,879 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:23,879 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:23,900 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/810eeec11f414b138733f9dc5e5ab56d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/810eeec11f414b138733f9dc5e5ab56d 2023-04-18 18:56:23,905 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c48561c851d33e29da879cafb393a8fd/B of c48561c851d33e29da879cafb393a8fd into 810eeec11f414b138733f9dc5e5ab56d(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:23,905 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:23,905 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/B, priority=11, startTime=1681844183851; duration=0sec 2023-04-18 18:56:23,906 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:23,906 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:56:23,907 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59168 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:56:23,907 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/C is initiating minor compaction (all files) 2023-04-18 18:56:23,907 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/C in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:23,907 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/2cccb0f4958647be918e040091173531, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/adf1516194914331acf01e7b77f867b8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/c1b8c3370ad7424896b725be9bc1d275, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/02b687b02f5c4c23b60c1f57178c20a0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/c42d6373e1db44d28ee87ad35cd14ce6] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=57.8 K 2023-04-18 18:56:23,919 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 2cccb0f4958647be918e040091173531, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681844180831 2023-04-18 18:56:23,920 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting adf1516194914331acf01e7b77f867b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1681844180883 2023-04-18 18:56:23,920 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting c1b8c3370ad7424896b725be9bc1d275, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681844180917 2023-04-18 18:56:23,921 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 02b687b02f5c4c23b60c1f57178c20a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1681844182053 2023-04-18 18:56:23,921 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting c42d6373e1db44d28ee87ad35cd14ce6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1681844183181 2023-04-18 18:56:23,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:23,923 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:23,925 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304187041fc2efc0448d1b1ea54815ada3135 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304187041fc2efc0448d1b1ea54815ada3135 2023-04-18 18:56:23,936 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=410, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/0a93a96a16464f3cb20179daa3ce95d7 2023-04-18 18:56:23,938 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#C#compaction#623 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:23,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:23,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844243935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:23,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844243936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:23,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844243938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,951 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/e85d10329d024b51929b0e9fb0c394a6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/e85d10329d024b51929b0e9fb0c394a6 2023-04-18 18:56:23,952 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/690a9ba04de543edaa9b0ba151f376e1 2023-04-18 18:56:23,956 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c48561c851d33e29da879cafb393a8fd/C of c48561c851d33e29da879cafb393a8fd into e85d10329d024b51929b0e9fb0c394a6(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:23,957 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:23,957 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/C, priority=11, startTime=1681844183851; duration=0sec 2023-04-18 18:56:23,957 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:23,965 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/1c61ce9185174b8395165f14858bdaf2 2023-04-18 18:56:23,971 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/0a93a96a16464f3cb20179daa3ce95d7 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/0a93a96a16464f3cb20179daa3ce95d7 2023-04-18 18:56:23,974 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/0a93a96a16464f3cb20179daa3ce95d7, entries=150, sequenceid=410, filesize=25.4 K 2023-04-18 18:56:23,975 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/690a9ba04de543edaa9b0ba151f376e1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/690a9ba04de543edaa9b0ba151f376e1 2023-04-18 18:56:23,979 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/690a9ba04de543edaa9b0ba151f376e1, entries=150, sequenceid=410, filesize=11.9 K 2023-04-18 18:56:23,980 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/1c61ce9185174b8395165f14858bdaf2 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/1c61ce9185174b8395165f14858bdaf2 2023-04-18 18:56:23,984 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/1c61ce9185174b8395165f14858bdaf2, entries=150, sequenceid=410, filesize=11.9 K 2023-04-18 18:56:23,984 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for c48561c851d33e29da879cafb393a8fd in 105ms, sequenceid=410, compaction requested=false 2023-04-18 18:56:23,984 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:23,984 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:23,985 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:23,985 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:23,985 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,985 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:23,985 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:23,987 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,988 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:23,988 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:23,988 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:23,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:23,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:23,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:23,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16d168a1[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:23,989 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:23,989 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:23,990 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:23,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:23,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:23,991 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:23,991 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:23,991 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:23,991 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,991 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:23,992 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,992 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:23,992 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:23,992 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:23,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:23,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:23,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:23,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:23,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:23,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:23,994 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:23,994 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:23,994 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:23,994 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:23,994 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:23,996 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:23,996 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:23,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:23,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:23,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:23,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:23,996 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:23,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:23,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:23,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:23,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:23,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:23,997 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:23,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:23,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:23,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,998 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:23,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:23,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:23,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,001 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,001 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,001 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,001 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:24,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:24,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:24,001 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,001 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:24,001 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:24,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:24,001 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:24,001 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,001 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:24,002 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,001 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:24,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:24,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:24,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:24,042 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-18 18:56:24,043 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:24,043 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:24,043 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:24,043 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:24,043 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:24,043 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:24,054 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304181a405c55378d459d88546852ab059065 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304181a405c55378d459d88546852ab059065 2023-04-18 18:56:24,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:24,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844244062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:24,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844244063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:24,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844244064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,102 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:24,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:24,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:24,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:24,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:24,105 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:24,105 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:24,105 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:24,105 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:24,106 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,106 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,107 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,107 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:24,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:24,107 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,107 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:24,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:24,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:24,109 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:24,109 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:24,109 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:24,109 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:24,111 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,111 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:24,111 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,111 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:24,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:24,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65bfedb6[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:24,113 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:24,113 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,114 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,114 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:24,114 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,115 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:24,115 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:24,115 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:24,117 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,117 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:24,117 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:24,117 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:24,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:24,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:24,120 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:24,120 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:24,120 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:24,120 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:24,120 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:24,124 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,124 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:24,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:24,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,124 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:24,125 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:24,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:24,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:24,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:24,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,126 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:24,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,128 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:24,128 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:24,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:24,129 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,129 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:24,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:24,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:24,129 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:24,129 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:24,129 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,129 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:24,129 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,129 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:24,129 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:24,129 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:24,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:24,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844244166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:24,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844244167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:24,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844244167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,229 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:24,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:24,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:24,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:24,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:24,233 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:24,233 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:24,233 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:24,233 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:24,233 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,233 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,234 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:24,234 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:24,235 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,235 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:24,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:24,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:24,236 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:24,236 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:24,236 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:24,236 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:24,236 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:24,236 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:24,236 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:24,237 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:24,237 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:24,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:24,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:24,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:24,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:24,239 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,239 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:24,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:24,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:24,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,240 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:24,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:24,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:24,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:24,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@723eb395[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:24,241 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:24,241 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,243 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,243 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:24,243 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:24,243 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:24,243 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,243 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:24,245 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,245 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:24,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:24,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:24,245 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:24,245 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:24,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:24,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:24,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:24,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:24,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:24,247 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:24,247 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:24,247 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:24,247 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:24,247 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:24,249 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,249 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:24,249 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:24,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:24,250 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:24,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,251 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:24,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,254 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:24,254 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:24,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:24,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:24,254 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:24,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:24,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:24,255 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:24,255 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:24,254 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:24,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:24,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,311 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/e04f88ce51b14d45b5f879efd88d1445 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/e04f88ce51b14d45b5f879efd88d1445 2023-04-18 18:56:24,315 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c48561c851d33e29da879cafb393a8fd/A of c48561c851d33e29da879cafb393a8fd into e04f88ce51b14d45b5f879efd88d1445(size=26.3 K), total size for store is 51.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:24,315 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:24,315 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/A, priority=11, startTime=1681844183851; duration=0sec 2023-04-18 18:56:24,316 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:24,355 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:24,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:24,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:24,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:24,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:24,358 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:24,359 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:24,359 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:24,359 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:24,359 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,359 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,360 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:24,360 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:24,361 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,361 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:24,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:24,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:24,362 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:24,362 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:24,362 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:24,362 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:24,362 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:24,362 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:24,362 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:24,362 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:24,362 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:24,362 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:24,362 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:24,362 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,362 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:24,362 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:24,364 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,364 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:24,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:24,364 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,364 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:24,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:24,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:24,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:24,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:24,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44ab797[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:24,366 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:24,367 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,368 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,368 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:24,368 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:24,368 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,368 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:24,368 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:24,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:24,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844244369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,371 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,371 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:24,371 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:24,371 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:24,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:24,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:24,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:24,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:24,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:24,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:24,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844244370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:24,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:24,373 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:24,373 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:24,373 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:24,373 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:24,374 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:24,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:24,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844244371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,375 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,375 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,375 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:24,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:24,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:24,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:24,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:24,376 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:24,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:24,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,378 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:24,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,379 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,380 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:24,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:24,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:24,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:24,380 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:24,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:24,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:24,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:24,380 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:24,380 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:24,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:24,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:24,459 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=429, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/3dc0f98887064e4cb31100d4ce47c7b9 2023-04-18 18:56:24,468 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/9944eb7c7382437194b8384133328f7e 2023-04-18 18:56:24,477 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/d936dd94bf0a40e9a4f15e21613ff15f 2023-04-18 18:56:24,480 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:24,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:24,481 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/3dc0f98887064e4cb31100d4ce47c7b9 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/3dc0f98887064e4cb31100d4ce47c7b9 2023-04-18 18:56:24,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:24,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:24,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:24,485 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:24,485 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:24,485 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:24,485 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:24,485 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,485 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,485 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/3dc0f98887064e4cb31100d4ce47c7b9, entries=200, sequenceid=429, filesize=32.3 K 2023-04-18 18:56:24,486 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/9944eb7c7382437194b8384133328f7e as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/9944eb7c7382437194b8384133328f7e 2023-04-18 18:56:24,486 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:24,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:24,487 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,487 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,487 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:24,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:24,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:24,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:24,488 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:24,488 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:24,488 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:24,488 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:24,488 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:24,488 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:24,488 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:24,488 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:24,488 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:24,488 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:24,488 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:24,488 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,488 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:24,488 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:24,491 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,491 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:24,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:24,491 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:24,491 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/9944eb7c7382437194b8384133328f7e, entries=150, sequenceid=429, filesize=11.9 K 2023-04-18 18:56:24,491 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:24,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:24,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,492 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/d936dd94bf0a40e9a4f15e21613ff15f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/d936dd94bf0a40e9a4f15e21613ff15f 2023-04-18 18:56:24,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:24,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:24,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19c353df[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:24,493 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:24,493 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,494 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,494 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:24,494 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:24,494 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:24,495 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,495 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:24,495 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,496 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/d936dd94bf0a40e9a4f15e21613ff15f, entries=150, sequenceid=429, filesize=11.9 K 2023-04-18 18:56:24,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:24,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:24,496 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:24,496 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:24,496 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:24,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:24,496 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for c48561c851d33e29da879cafb393a8fd in 454ms, sequenceid=429, compaction requested=true 2023-04-18 18:56:24,496 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:24,496 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:24,496 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:24,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:24,497 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:24,496 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:24,497 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:24,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,497 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86030 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:24,497 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/A is initiating minor compaction (all files) 2023-04-18 18:56:24,498 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/A in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:24,498 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:24,498 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/e04f88ce51b14d45b5f879efd88d1445, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/0a93a96a16464f3cb20179daa3ce95d7, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/3dc0f98887064e4cb31100d4ce47c7b9] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=84.0 K 2023-04-18 18:56:24,498 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/B is initiating minor compaction (all files) 2023-04-18 18:56:24,498 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/B in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:24,498 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/810eeec11f414b138733f9dc5e5ab56d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/690a9ba04de543edaa9b0ba151f376e1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/9944eb7c7382437194b8384133328f7e] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=36.5 K 2023-04-18 18:56:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:24,498 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting e04f88ce51b14d45b5f879efd88d1445, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1681844183181 2023-04-18 18:56:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,498 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 810eeec11f414b138733f9dc5e5ab56d, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1681844183181 2023-04-18 18:56:24,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,499 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 0a93a96a16464f3cb20179daa3ce95d7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1681844183813 2023-04-18 18:56:24,499 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 690a9ba04de543edaa9b0ba151f376e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1681844183813 2023-04-18 18:56:24,499 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 3dc0f98887064e4cb31100d4ce47c7b9, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1681844183936 2023-04-18 18:56:24,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:24,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:24,499 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:24,499 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:24,499 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:24,499 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:24,499 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:24,499 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 9944eb7c7382437194b8384133328f7e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1681844183936 2023-04-18 18:56:24,502 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,502 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:24,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,502 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:24,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:24,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:24,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:24,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:24,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,503 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:24,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,505 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:24,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,507 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#B#compaction#629 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:24,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,516 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,516 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,516 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,516 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,516 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,516 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:24,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:24,516 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:24,516 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:24,516 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,516 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:24,516 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:24,517 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:24,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:24,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:24,521 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#A#compaction#630 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:24,526 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/13bd1548f83f4b698345260613a0d29c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/13bd1548f83f4b698345260613a0d29c 2023-04-18 18:56:24,531 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c48561c851d33e29da879cafb393a8fd/B of c48561c851d33e29da879cafb393a8fd into 13bd1548f83f4b698345260613a0d29c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:24,531 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:24,531 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/B, priority=13, startTime=1681844184496; duration=0sec 2023-04-18 18:56:24,531 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:24,531 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:24,532 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:24,532 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/C is initiating minor compaction (all files) 2023-04-18 18:56:24,532 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/C in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:24,532 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/e85d10329d024b51929b0e9fb0c394a6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/1c61ce9185174b8395165f14858bdaf2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/d936dd94bf0a40e9a4f15e21613ff15f] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=36.5 K 2023-04-18 18:56:24,533 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting e85d10329d024b51929b0e9fb0c394a6, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1681844183181 2023-04-18 18:56:24,533 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 1c61ce9185174b8395165f14858bdaf2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1681844183813 2023-04-18 18:56:24,533 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting d936dd94bf0a40e9a4f15e21613ff15f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1681844183936 2023-04-18 18:56:24,537 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#C#compaction#631 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:24,617 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:24,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:24,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:24,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:24,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:24,620 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:24,620 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:24,620 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:24,620 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:24,621 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,621 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,622 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:24,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:24,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:24,622 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,623 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,623 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:24,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:24,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:24,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:24,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:24,624 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:24,624 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:24,624 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:24,624 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:24,624 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:24,625 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:24,625 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:24,625 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-18 18:56:24,625 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:24,625 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:24,625 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:24,625 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:24,625 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:24,625 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:24,646 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041854ae8bf4d2714a40bf072ead383ae8a6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041854ae8bf4d2714a40bf072ead383ae8a6 2023-04-18 18:56:24,653 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=446, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/4d1c48c367444ea4bc66ccf05825e9f2 2023-04-18 18:56:24,664 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/d085ea0e640f4e25943a48821501665f 2023-04-18 18:56:24,675 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:24,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:24,675 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/95dcc9fe111744dba6204d5e71117f8d 2023-04-18 18:56:24,679 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/4d1c48c367444ea4bc66ccf05825e9f2 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/4d1c48c367444ea4bc66ccf05825e9f2 2023-04-18 18:56:24,683 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/4d1c48c367444ea4bc66ccf05825e9f2, entries=150, sequenceid=446, filesize=25.4 K 2023-04-18 18:56:24,684 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/d085ea0e640f4e25943a48821501665f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/d085ea0e640f4e25943a48821501665f 2023-04-18 18:56:24,687 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/d085ea0e640f4e25943a48821501665f, entries=150, sequenceid=446, filesize=11.9 K 2023-04-18 18:56:24,688 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/95dcc9fe111744dba6204d5e71117f8d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/95dcc9fe111744dba6204d5e71117f8d 2023-04-18 18:56:24,692 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/95dcc9fe111744dba6204d5e71117f8d, entries=150, sequenceid=446, filesize=11.9 K 2023-04-18 18:56:24,692 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=87.22 KB/89310 for c48561c851d33e29da879cafb393a8fd in 67ms, sequenceid=446, compaction requested=false 2023-04-18 18:56:24,692 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:24,692 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:24,692 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:24,692 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:24,692 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,692 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:24,693 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:24,694 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:24,694 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:24,695 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-18 18:56:24,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:24,695 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,695 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:24,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:24,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:24,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:24,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:24,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:24,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:24,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:24,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:24,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:24,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38bcf0c[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:24,701 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:24,701 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,708 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,708 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:24,708 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,708 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:24,708 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:24,708 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:24,710 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:24,710 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:24,710 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,710 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:24,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:24,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:24,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:24,713 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:24,713 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:24,713 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:24,713 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:24,713 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:24,714 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,715 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,715 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:24,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:24,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:24,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:24,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:24,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:24,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,716 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:24,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844244713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:24,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:24,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844244713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:24,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,718 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:24,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844244717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,721 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,721 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,721 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,721 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,721 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:24,721 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:24,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:24,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:24,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:24,721 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:24,721 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:24,722 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,722 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,722 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:24,722 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:24,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:24,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:24,732 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418ecdbe72e318649968b2f02295b6b3e51 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418ecdbe72e318649968b2f02295b6b3e51 2023-04-18 18:56:24,745 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=464, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/47857ad268b64f15a8779a02b7fd669f 2023-04-18 18:56:24,753 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/bd4ce00e084247c1b4b6c9b66f05ac3b 2023-04-18 18:56:24,768 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/824b76f7867647e781a3dcaf8879d56a 2023-04-18 18:56:24,771 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/47857ad268b64f15a8779a02b7fd669f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/47857ad268b64f15a8779a02b7fd669f 2023-04-18 18:56:24,774 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/47857ad268b64f15a8779a02b7fd669f, entries=200, sequenceid=464, filesize=32.3 K 2023-04-18 18:56:24,775 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/bd4ce00e084247c1b4b6c9b66f05ac3b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/bd4ce00e084247c1b4b6c9b66f05ac3b 2023-04-18 18:56:24,778 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/bd4ce00e084247c1b4b6c9b66f05ac3b, entries=150, sequenceid=464, filesize=11.9 K 2023-04-18 18:56:24,778 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/824b76f7867647e781a3dcaf8879d56a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/824b76f7867647e781a3dcaf8879d56a 2023-04-18 18:56:24,781 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/824b76f7867647e781a3dcaf8879d56a, entries=150, sequenceid=464, filesize=11.9 K 2023-04-18 18:56:24,782 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for c48561c851d33e29da879cafb393a8fd in 87ms, sequenceid=464, compaction requested=true 2023-04-18 18:56:24,782 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:24,782 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:24,782 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-18 18:56:24,782 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-18 18:56:24,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:24,821 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-18 18:56:24,821 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:24,822 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:24,822 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:24,822 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:24,822 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:24,822 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:24,822 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:24,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:24,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:24,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:24,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:24,827 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:24,827 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:24,827 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:24,827 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:24,828 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,828 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,830 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:24,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:24,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:24,831 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,831 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:24,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:24,832 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:24,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:24,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:24,832 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:24,832 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:24,832 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:24,832 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:24,832 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:24,832 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:24,832 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:24,832 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:24,833 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:24,833 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:24,833 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:24,833 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,833 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:24,833 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:24,834 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,834 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418dfc3d4d4f067466c802fcc36e9f0ea5a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418dfc3d4d4f067466c802fcc36e9f0ea5a 2023-04-18 18:56:24,834 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,835 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:24,835 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:24,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:24,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:24,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ae76f3[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:24,838 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:24,838 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,839 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,839 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:24,840 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:24,840 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:24,840 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=480, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/169bbddb8445404b985204c4c2c41f1f 2023-04-18 18:56:24,840 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,840 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:24,842 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,842 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:24,842 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:24,842 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:24,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:24,845 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:24,845 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:24,845 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:24,845 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:24,845 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:24,848 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:24,848 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844244843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,848 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:24,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:24,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:24,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844244844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:24,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:24,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:24,850 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:24,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,851 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:24,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:24,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844244848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,853 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:24,853 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:24,853 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,853 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:24,853 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:24,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:24,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:24,854 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,854 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,854 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:24,854 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:24,854 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,854 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,854 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:24,855 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/b74aa7b83cc043f19dceba3a360837d0 2023-04-18 18:56:24,950 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/8bcd54fc71bb45a3823a030401d2a9eb as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/8bcd54fc71bb45a3823a030401d2a9eb 2023-04-18 18:56:24,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:24,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844244950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:24,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43936 deadline: 1681844244950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,954 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:24,954 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c48561c851d33e29da879cafb393a8fd/C of c48561c851d33e29da879cafb393a8fd into 8bcd54fc71bb45a3823a030401d2a9eb(size=12.8 K), total size for store is 36.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:24,954 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:24,954 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/C, priority=13, startTime=1681844184497; duration=0sec 2023-04-18 18:56:24,954 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-18 18:56:24,955 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-04-18 18:56:24,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:24,955 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:24,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844244954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,956 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-18 18:56:24,956 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-18 18:56:24,956 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. because compaction request was cancelled 2023-04-18 18:56:24,956 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:24,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:24,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:24,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:24,958 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:24,958 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:24,958 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:24,958 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:24,958 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,958 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,958 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:24,959 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/B is initiating minor compaction (all files) 2023-04-18 18:56:24,959 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/B in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:24,959 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/13bd1548f83f4b698345260613a0d29c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/d085ea0e640f4e25943a48821501665f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/bd4ce00e084247c1b4b6c9b66f05ac3b] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=36.6 K 2023-04-18 18:56:24,959 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 13bd1548f83f4b698345260613a0d29c, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1681844183936 2023-04-18 18:56:24,960 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,960 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/212e3d596b6a4111b3e0a257ab8ccde6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/212e3d596b6a4111b3e0a257ab8ccde6 2023-04-18 18:56:24,960 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:24,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:24,960 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,960 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,961 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting d085ea0e640f4e25943a48821501665f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1681844184061 2023-04-18 18:56:24,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:24,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:24,961 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting bd4ce00e084247c1b4b6c9b66f05ac3b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1681844184690 2023-04-18 18:56:24,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:24,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:24,962 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:24,962 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:24,962 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:24,962 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:24,962 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:24,962 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:24,966 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:24,966 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:24,966 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:24,966 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:24,966 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:24,966 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,966 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:24,966 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:24,968 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,968 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:24,969 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c48561c851d33e29da879cafb393a8fd/A of c48561c851d33e29da879cafb393a8fd into 212e3d596b6a4111b3e0a257ab8ccde6(size=26.4 K), total size for store is 84.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:24,969 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:24,969 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/A, priority=13, startTime=1681844184496; duration=0sec 2023-04-18 18:56:24,969 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:24,969 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 18:56:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:24,969 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,969 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,970 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 18:56:24,970 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/C is initiating minor compaction (all files) 2023-04-18 18:56:24,970 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/C in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:24,970 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/8bcd54fc71bb45a3823a030401d2a9eb, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/95dcc9fe111744dba6204d5e71117f8d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/824b76f7867647e781a3dcaf8879d56a] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=36.6 K 2023-04-18 18:56:24,972 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 8bcd54fc71bb45a3823a030401d2a9eb, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1681844183936 2023-04-18 18:56:24,972 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 95dcc9fe111744dba6204d5e71117f8d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1681844184061 2023-04-18 18:56:24,973 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 824b76f7867647e781a3dcaf8879d56a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1681844184690 2023-04-18 18:56:24,973 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#B#compaction#641 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:24,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:24,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:24,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55899ffe[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:24,977 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:24,977 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,980 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:24,980 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:24,980 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:24,980 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:24,980 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,980 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:24,982 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:24,982 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:24,982 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,982 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:24,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:24,984 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:24,985 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:24,985 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:24,985 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:24,985 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:24,987 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,987 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:24,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,987 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:24,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:24,988 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,989 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:24,993 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#C#compaction#642 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:24,996 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/7831a5574d7448818a75dfc827259c7b as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/7831a5574d7448818a75dfc827259c7b 2023-04-18 18:56:25,000 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c48561c851d33e29da879cafb393a8fd/B of c48561c851d33e29da879cafb393a8fd into 7831a5574d7448818a75dfc827259c7b(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:25,000 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:25,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:25,000 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:25,000 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:25,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:25,000 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:25,000 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:25,000 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:25,000 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/B, priority=13, startTime=1681844184782; duration=0sec 2023-04-18 18:56:25,000 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:25,000 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:25,001 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,001 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:25,001 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:25,001 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:25,000 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:25,001 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:25,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:25,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:25,101 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:25,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:25,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:25,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:25,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:25,105 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:25,105 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:25,105 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:25,105 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:25,105 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:25,105 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:25,106 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,106 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:25,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:25,107 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,107 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:25,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:25,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:25,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:25,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:25,109 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:25,109 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:25,109 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:25,109 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:25,109 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:25,109 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:25,109 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:25,109 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:25,109 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,109 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:25,109 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:25,111 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:25,111 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:25,111 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:25,111 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:25,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:25,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5657633b[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:25,112 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:25,113 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:25,115 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:25,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:25,115 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:25,115 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:25,115 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,115 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:25,115 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:25,116 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,116 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:25,116 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:25,116 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:25,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:25,119 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:25,119 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:25,119 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:25,119 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:25,119 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:25,121 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:25,121 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:25,121 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:25,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:25,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:25,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:25,122 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:25,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:25,123 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:25,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,126 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,126 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:25,126 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:25,126 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:25,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:25,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:25,126 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:25,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:25,127 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:25,126 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:25,127 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:25,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:25,127 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,127 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:25,127 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:25,127 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:25,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:25,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:25,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844245154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:25,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844245154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:25,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844245157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,227 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:25,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:25,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:25,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:25,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:25,230 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:25,230 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:25,230 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:25,230 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:25,230 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:25,231 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:25,233 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:25,233 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:25,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:25,233 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,233 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:25,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:25,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:25,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:25,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:25,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:25,234 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:25,234 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:25,234 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:25,234 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:25,234 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:25,234 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:25,235 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:25,235 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:25,235 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:25,235 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:25,235 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:25,235 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,235 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:25,235 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:25,236 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,236 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:25,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:25,237 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:25,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:25,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bf92552[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:25,238 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:25,238 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:25,239 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:25,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:25,239 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:25,239 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:25,239 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:25,239 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,239 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:25,241 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,241 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:25,241 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:25,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:25,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:25,241 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:25,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:25,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:25,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:25,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:25,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:25,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:25,243 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:25,243 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:25,243 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:25,243 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:25,243 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:25,246 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:25,246 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:25,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:25,246 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:25,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:25,247 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,249 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:25,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:25,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:25,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:25,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:25,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:25,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:25,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:25,255 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:25,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:25,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:25,256 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:25,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:25,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:25,257 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:25,257 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:25,263 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/f5f8b01ec6cd4a4fada91962c850343c 2023-04-18 18:56:25,267 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/169bbddb8445404b985204c4c2c41f1f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/169bbddb8445404b985204c4c2c41f1f 2023-04-18 18:56:25,270 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/169bbddb8445404b985204c4c2c41f1f, entries=250, sequenceid=480, filesize=39.2 K 2023-04-18 18:56:25,271 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/b74aa7b83cc043f19dceba3a360837d0 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b74aa7b83cc043f19dceba3a360837d0 2023-04-18 18:56:25,274 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b74aa7b83cc043f19dceba3a360837d0, entries=150, sequenceid=480, filesize=11.9 K 2023-04-18 18:56:25,275 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/f5f8b01ec6cd4a4fada91962c850343c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/f5f8b01ec6cd4a4fada91962c850343c 2023-04-18 18:56:25,278 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/f5f8b01ec6cd4a4fada91962c850343c, entries=150, sequenceid=480, filesize=11.9 K 2023-04-18 18:56:25,278 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for c48561c851d33e29da879cafb393a8fd in 457ms, sequenceid=480, compaction requested=true 2023-04-18 18:56:25,278 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:25,279 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:25,279 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:25,279 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-18 18:56:25,279 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:56:25,280 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 126262 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:56:25,280 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/A is initiating minor compaction (all files) 2023-04-18 18:56:25,280 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/A in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:25,280 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/212e3d596b6a4111b3e0a257ab8ccde6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/4d1c48c367444ea4bc66ccf05825e9f2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/47857ad268b64f15a8779a02b7fd669f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/169bbddb8445404b985204c4c2c41f1f] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=123.3 K 2023-04-18 18:56:25,280 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 212e3d596b6a4111b3e0a257ab8ccde6, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1681844183936 2023-04-18 18:56:25,281 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 4d1c48c367444ea4bc66ccf05825e9f2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1681844184061 2023-04-18 18:56:25,281 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 47857ad268b64f15a8779a02b7fd669f, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1681844184690 2023-04-18 18:56:25,281 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 169bbddb8445404b985204c4c2c41f1f, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1681844184710 2023-04-18 18:56:25,289 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#A#compaction#643 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:25,357 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:25,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:25,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:25,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:25,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:25,360 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:25,360 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:25,360 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:25,360 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:25,361 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:25,361 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:25,362 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,362 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:25,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:25,362 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,362 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:25,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:25,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:25,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:25,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:25,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:25,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:25,363 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:25,365 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:25,365 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:25,365 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:25,365 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:25,365 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-18 18:56:25,366 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:25,366 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:25,366 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:25,366 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:25,366 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:25,366 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:25,410 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/7530f5b937cd401a8c6e3caf477853c1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/7530f5b937cd401a8c6e3caf477853c1 2023-04-18 18:56:25,413 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c48561c851d33e29da879cafb393a8fd/C of c48561c851d33e29da879cafb393a8fd into 7530f5b937cd401a8c6e3caf477853c1(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:25,413 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:25,413 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/C, priority=13, startTime=1681844184782; duration=0sec 2023-04-18 18:56:25,414 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-18 18:56:25,414 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-18 18:56:25,414 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-18 18:56:25,414 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-18 18:56:25,414 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. because compaction request was cancelled 2023-04-18 18:56:25,414 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-18 18:56:25,415 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-18 18:56:25,415 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-18 18:56:25,415 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. because compaction request was cancelled 2023-04-18 18:56:25,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:25,458 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:25,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:25,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844245470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:25,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844245471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:25,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844245473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:25,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844245574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:25,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844245575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:25,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844245576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:25,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844245777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:25,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844245778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:25,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844245780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:25,786 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041849dc21f71f6e465ebed1ea015cb0f5a5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041849dc21f71f6e465ebed1ea015cb0f5a5 2023-04-18 18:56:25,790 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=499, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/2779ebd33a3f4b44ad9551ab5d50eeb6 2023-04-18 18:56:26,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:26,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:26,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844246082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844246083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:26,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844246085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,108 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/ff40a39f7a994c4ab96c668d3c6950ac as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/ff40a39f7a994c4ab96c668d3c6950ac 2023-04-18 18:56:26,111 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c48561c851d33e29da879cafb393a8fd/A of c48561c851d33e29da879cafb393a8fd into ff40a39f7a994c4ab96c668d3c6950ac(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:26,111 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:26,111 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/A, priority=12, startTime=1681844185278; duration=0sec 2023-04-18 18:56:26,112 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:26,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:26,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43904 deadline: 1681844246113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,115 DEBUG [Thread-3979] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:56:26,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:26,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43862 deadline: 1681844246129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,133 DEBUG [Thread-3973] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8176 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., hostname=jenkins-hbase4.apache.org,34053,1681844018952, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 18:56:26,199 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/fafad7032e624390aa39cc48ff46e8c8 2023-04-18 18:56:26,226 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/2413b994d66b4c3597e592909fd714c4 2023-04-18 18:56:26,230 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/2779ebd33a3f4b44ad9551ab5d50eeb6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/2779ebd33a3f4b44ad9551ab5d50eeb6 2023-04-18 18:56:26,234 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/2779ebd33a3f4b44ad9551ab5d50eeb6, entries=150, sequenceid=499, filesize=25.4 K 2023-04-18 18:56:26,235 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/fafad7032e624390aa39cc48ff46e8c8 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fafad7032e624390aa39cc48ff46e8c8 2023-04-18 18:56:26,239 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fafad7032e624390aa39cc48ff46e8c8, entries=150, sequenceid=499, filesize=11.9 K 2023-04-18 18:56:26,239 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/2413b994d66b4c3597e592909fd714c4 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/2413b994d66b4c3597e592909fd714c4 2023-04-18 18:56:26,242 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/2413b994d66b4c3597e592909fd714c4, entries=150, sequenceid=499, filesize=11.9 K 2023-04-18 18:56:26,244 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for c48561c851d33e29da879cafb393a8fd in 879ms, sequenceid=499, compaction requested=true 2023-04-18 18:56:26,244 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:26,244 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:26,244 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:26,244 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:26,244 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,244 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:26,244 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:26,245 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,245 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:26,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:26,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:26,246 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:26,246 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:26,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:26,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c17c330[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:26,248 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:26,248 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:26,250 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:26,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:26,250 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:26,250 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:26,250 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:26,251 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,251 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:26,252 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:26,252 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:26,252 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,252 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:26,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:26,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:26,254 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:26,254 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:26,254 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:26,255 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:26,255 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:26,257 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:26,257 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:26,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:26,257 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:26,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:26,259 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,260 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,262 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:26,262 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:26,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:26,262 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,263 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:26,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:26,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:26,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:26,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:26,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:26,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:26,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:26,263 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:26,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:26,263 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:26,263 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:26,363 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:26,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:26,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:26,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:26,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:26,367 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:26,367 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:26,367 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:26,367 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:26,367 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:26,367 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:26,369 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:26,369 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:26,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:26,370 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,370 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:26,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:26,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:26,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:26,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:26,371 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:26,371 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:26,371 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:26,371 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:26,371 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:26,371 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:26,371 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:26,371 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-18 18:56:26,372 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:26,372 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:26,372 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:26,372 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:26,372 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:26,372 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:26,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:26,590 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:26,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:26,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844246606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:26,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844246606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:26,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844246607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:26,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844246710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:26,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844246711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:26,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844246711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,789 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041813904312c90149baaeb71cb34c77e280 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041813904312c90149baaeb71cb34c77e280 2023-04-18 18:56:26,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:26,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844246912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:26,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:26,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844246915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:26,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844246915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,196 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=517, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/7b0c1cd7b76d479d84b93997422e4c2c 2023-04-18 18:56:27,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:27,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844247217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:27,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844247218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:27,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844247220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,606 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/fc3bc2f048d94a8cb9ed0b9b1deed10d 2023-04-18 18:56:27,616 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/44abe87c5766411dbb33020552637632 2023-04-18 18:56:27,621 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/7b0c1cd7b76d479d84b93997422e4c2c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/7b0c1cd7b76d479d84b93997422e4c2c 2023-04-18 18:56:27,625 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/7b0c1cd7b76d479d84b93997422e4c2c, entries=150, sequenceid=517, filesize=25.4 K 2023-04-18 18:56:27,626 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/fc3bc2f048d94a8cb9ed0b9b1deed10d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fc3bc2f048d94a8cb9ed0b9b1deed10d 2023-04-18 18:56:27,629 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fc3bc2f048d94a8cb9ed0b9b1deed10d, entries=150, sequenceid=517, filesize=11.9 K 2023-04-18 18:56:27,630 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/44abe87c5766411dbb33020552637632 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/44abe87c5766411dbb33020552637632 2023-04-18 18:56:27,633 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/44abe87c5766411dbb33020552637632, entries=150, sequenceid=517, filesize=11.9 K 2023-04-18 18:56:27,634 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for c48561c851d33e29da879cafb393a8fd in 1263ms, sequenceid=517, compaction requested=true 2023-04-18 18:56:27,634 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:27,634 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:27,634 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:27,634 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:27,634 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,634 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:27,634 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:27,639 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,639 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:27,639 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,639 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:27,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:27,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6578d523[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:27,641 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:27,641 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,643 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:27,643 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,643 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:27,645 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,645 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:27,645 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:27,645 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:27,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:27,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:27,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:27,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:27,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:27,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:27,648 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:27,648 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:27,648 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:27,648 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:27,648 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:27,651 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,651 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:27,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,652 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:27,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,652 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:27,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:27,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:27,654 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,656 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,656 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:27,656 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:27,656 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:27,656 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,656 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,656 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,656 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,656 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:27,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:27,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:27,657 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:27,657 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:27,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:27,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:27,656 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:27,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:27,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:27,725 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-18 18:56:27,725 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:27,725 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:27,725 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:27,725 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:27,725 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:27,725 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:27,737 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418b92d7ab12ee1414594747ca7e8550255 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418b92d7ab12ee1414594747ca7e8550255 2023-04-18 18:56:27,741 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=535, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/35d1cdfee9344e0bb9b82a603fe41410 2023-04-18 18:56:27,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:27,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844247741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:27,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844247746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:27,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844247747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,757 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:27,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:27,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:27,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:27,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:27,760 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:27,760 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:27,760 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:27,760 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:27,760 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,760 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:27,761 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:27,761 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:27,762 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,762 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:27,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:27,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:27,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:27,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:27,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:27,763 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:27,763 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:27,763 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:27,763 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:27,763 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:27,763 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:27,763 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:27,763 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:27,763 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:27,764 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:27,764 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:27,764 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,764 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:27,764 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:27,765 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,765 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:27,765 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,765 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:27,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:27,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:27,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:27,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:27,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:27,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@85b2c0c[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:27,768 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:27,768 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,769 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,770 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:27,770 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:27,770 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:27,770 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,770 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:27,771 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:27,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:27,771 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:27,771 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:27,771 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:27,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:27,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:27,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:27,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:27,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:27,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:27,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:27,774 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:27,774 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:27,774 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:27,774 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:27,774 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:27,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,777 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:27,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,777 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:27,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:27,777 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:27,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:27,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:27,778 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:27,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:27,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,781 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,781 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:27,781 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:27,781 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:27,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:27,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:27,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:27,781 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:27,781 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:27,781 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,781 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,781 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,781 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,781 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:27,782 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:27,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:27,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:27,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:27,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844247848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:27,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844247851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:27,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844247851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,882 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:27,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:27,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:27,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:27,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:27,885 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:27,885 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:27,885 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:27,885 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:27,886 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,886 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:27,888 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,888 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:27,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:27,889 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,889 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:27,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:27,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:27,890 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:27,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:27,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:27,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:27,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:27,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:27,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:27,890 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:27,890 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:27,890 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:27,891 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:27,891 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:27,891 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:27,891 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:27,891 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,891 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:27,891 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:27,892 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,892 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:27,893 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,893 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:27,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:27,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aca4be0[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:27,894 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:27,894 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,895 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,896 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:27,896 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:27,896 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:27,896 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,896 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:27,897 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:27,897 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:27,897 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,897 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:27,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:27,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:27,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:27,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:27,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:27,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:27,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:27,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:27,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:27,900 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:27,900 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:27,900 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:27,900 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:27,900 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:27,902 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,902 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,902 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:27,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:27,903 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:27,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:27,905 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:27,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,907 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:27,907 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:27,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:27,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:27,908 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,908 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:27,908 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:27,908 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:27,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:27,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:27,908 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,908 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:27,908 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:27,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:27,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:27,908 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:27,908 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:27,908 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:27,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:28,008 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:28,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:28,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:28,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:28,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:28,012 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:28,012 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:28,012 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:28,012 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:28,013 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,013 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:28,014 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:28,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:28,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:28,014 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:28,014 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,014 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:28,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:28,015 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:28,015 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:28,015 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:28,015 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:28,016 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:28,016 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:28,016 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:28,016 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:28,016 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:28,016 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:28,016 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:28,016 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:28,016 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:28,016 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:28,016 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,016 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:28,016 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:28,018 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:28,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:28,018 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:28,018 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,018 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:28,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:28,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:28,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:28,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:28,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8060fca[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:28,020 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:28,020 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,025 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,025 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:28,025 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:28,025 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:28,025 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,025 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:28,026 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:28,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:28,026 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:28,026 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:28,027 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:28,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:28,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:28,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:28,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:28,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:28,029 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:28,030 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:28,030 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:28,030 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:28,030 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:28,031 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,031 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:28,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,031 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:28,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:28,032 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,033 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,036 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:28,036 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:28,036 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,036 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:28,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:28,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:28,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:28,036 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:28,036 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:28,036 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:28,036 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:28,036 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,037 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:28,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:28,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844248050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:28,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844248053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:28,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844248054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,137 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:28,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:28,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:28,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:28,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:28,140 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:28,140 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:28,140 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:28,140 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:28,141 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,141 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:28,143 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:28,143 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:28,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:28,143 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:28,143 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:28,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:28,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:28,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:28,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:28,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:28,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:28,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:28,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:28,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:28,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:28,145 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:28,145 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:28,145 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:28,145 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:28,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:28,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:28,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:28,145 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:28,147 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,147 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:28,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:28,147 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,147 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:28,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:28,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:28,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:28,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:28,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:28,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b740224[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:28,149 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:28,149 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,150 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,150 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:28,150 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:28,150 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:28,150 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,150 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:28,150 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/8d4842bc2eba4954a890ddc4d9fb5076 2023-04-18 18:56:28,152 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:28,152 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,152 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:28,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:28,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:28,152 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:28,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:28,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:28,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:28,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:28,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:28,156 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:28,156 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:28,156 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:28,156 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:28,156 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:28,158 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,158 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:28,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,158 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:28,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:28,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:28,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:28,159 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:28,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,161 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:28,161 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/fc5c2e22964a4f8f9c7c29a30fb9b2b3 2023-04-18 18:56:28,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:28,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,165 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,165 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:28,165 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:28,165 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,165 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,165 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,165 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,165 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:28,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:28,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:28,165 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:28,166 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:28,166 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:28,166 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:28,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:28,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:28,166 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/35d1cdfee9344e0bb9b82a603fe41410 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/35d1cdfee9344e0bb9b82a603fe41410 2023-04-18 18:56:28,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:28,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:28,170 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/35d1cdfee9344e0bb9b82a603fe41410, entries=200, sequenceid=535, filesize=32.3 K 2023-04-18 18:56:28,170 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/8d4842bc2eba4954a890ddc4d9fb5076 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/8d4842bc2eba4954a890ddc4d9fb5076 2023-04-18 18:56:28,174 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/8d4842bc2eba4954a890ddc4d9fb5076, entries=150, sequenceid=535, filesize=11.9 K 2023-04-18 18:56:28,174 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/fc5c2e22964a4f8f9c7c29a30fb9b2b3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/fc5c2e22964a4f8f9c7c29a30fb9b2b3 2023-04-18 18:56:28,177 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/fc5c2e22964a4f8f9c7c29a30fb9b2b3, entries=150, sequenceid=535, filesize=11.9 K 2023-04-18 18:56:28,178 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for c48561c851d33e29da879cafb393a8fd in 453ms, sequenceid=535, compaction requested=true 2023-04-18 18:56:28,178 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:28,178 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:28,178 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:56:28,178 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:56:28,178 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:28,179 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:28,180 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 112314 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:56:28,180 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/A is initiating minor compaction (all files) 2023-04-18 18:56:28,180 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/A in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:28,180 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/ff40a39f7a994c4ab96c668d3c6950ac, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/2779ebd33a3f4b44ad9551ab5d50eeb6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/7b0c1cd7b76d479d84b93997422e4c2c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/35d1cdfee9344e0bb9b82a603fe41410] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=109.7 K 2023-04-18 18:56:28,181 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting ff40a39f7a994c4ab96c668d3c6950ac, keycount=150, bloomtype=ROW, size=26.5 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1681844184711 2023-04-18 18:56:28,181 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 2779ebd33a3f4b44ad9551ab5d50eeb6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1681844184836 2023-04-18 18:56:28,181 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61986 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:56:28,181 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/B is initiating minor compaction (all files) 2023-04-18 18:56:28,181 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/B in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:28,182 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/7831a5574d7448818a75dfc827259c7b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b74aa7b83cc043f19dceba3a360837d0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fafad7032e624390aa39cc48ff46e8c8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fc3bc2f048d94a8cb9ed0b9b1deed10d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/8d4842bc2eba4954a890ddc4d9fb5076] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=60.5 K 2023-04-18 18:56:28,182 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 7b0c1cd7b76d479d84b93997422e4c2c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1681844185467 2023-04-18 18:56:28,182 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 7831a5574d7448818a75dfc827259c7b, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1681844184690 2023-04-18 18:56:28,182 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 35d1cdfee9344e0bb9b82a603fe41410, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681844186605 2023-04-18 18:56:28,182 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b74aa7b83cc043f19dceba3a360837d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1681844184711 2023-04-18 18:56:28,183 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting fafad7032e624390aa39cc48ff46e8c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1681844184836 2023-04-18 18:56:28,184 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting fc3bc2f048d94a8cb9ed0b9b1deed10d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1681844185467 2023-04-18 18:56:28,184 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 8d4842bc2eba4954a890ddc4d9fb5076, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681844186605 2023-04-18 18:56:28,202 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#B#compaction#654 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:28,214 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#A#compaction#653 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:28,227 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/72cd48c9169a45bfa30eaeb3392fd66a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/72cd48c9169a45bfa30eaeb3392fd66a 2023-04-18 18:56:28,232 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c48561c851d33e29da879cafb393a8fd/B of c48561c851d33e29da879cafb393a8fd into 72cd48c9169a45bfa30eaeb3392fd66a(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:28,232 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:28,232 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/B, priority=11, startTime=1681844188178; duration=0sec 2023-04-18 18:56:28,232 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:28,232 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:56:28,234 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61986 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:56:28,234 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/C is initiating minor compaction (all files) 2023-04-18 18:56:28,234 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/C in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:28,234 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/7530f5b937cd401a8c6e3caf477853c1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/f5f8b01ec6cd4a4fada91962c850343c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/2413b994d66b4c3597e592909fd714c4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/44abe87c5766411dbb33020552637632, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/fc5c2e22964a4f8f9c7c29a30fb9b2b3] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=60.5 K 2023-04-18 18:56:28,234 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 7530f5b937cd401a8c6e3caf477853c1, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1681844184690 2023-04-18 18:56:28,235 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting f5f8b01ec6cd4a4fada91962c850343c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1681844184711 2023-04-18 18:56:28,235 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 2413b994d66b4c3597e592909fd714c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1681844184836 2023-04-18 18:56:28,235 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 44abe87c5766411dbb33020552637632, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1681844185467 2023-04-18 18:56:28,235 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting fc5c2e22964a4f8f9c7c29a30fb9b2b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681844186605 2023-04-18 18:56:28,242 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#C#compaction#655 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:28,253 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/461a005cc02442fcbe813ac9c2ca77f2 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/461a005cc02442fcbe813ac9c2ca77f2 2023-04-18 18:56:28,257 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c48561c851d33e29da879cafb393a8fd/C of c48561c851d33e29da879cafb393a8fd into 461a005cc02442fcbe813ac9c2ca77f2(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:28,258 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:28,258 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/C, priority=11, startTime=1681844188179; duration=0sec 2023-04-18 18:56:28,258 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:28,266 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:28,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:28,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:28,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:28,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:28,269 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:28,269 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:28,269 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:28,269 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:28,270 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,270 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:28,271 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:28,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:28,271 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:28,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,272 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:28,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:28,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:28,274 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:28,274 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:28,274 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:28,274 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:28,274 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:28,274 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:28,274 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:28,274 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:28,274 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:28,274 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-18 18:56:28,275 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:28,275 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:28,275 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:28,275 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:28,275 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:28,275 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:28,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:28,356 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:28,373 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:28,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844248370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:28,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844248372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:28,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844248373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:28,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844248474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:28,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844248475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:28,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844248476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:28,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844248676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:28,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844248677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:28,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844248679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,700 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418302fb6778ba04279a86b07ebd82f733c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418302fb6778ba04279a86b07ebd82f733c 2023-04-18 18:56:28,706 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=551, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/07e85023b877485485f207d2cbfce68d 2023-04-18 18:56:28,716 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/723c28dc9b8e403c9a0f8632b5e908a1 2023-04-18 18:56:28,725 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/4a97cdefdc7446e8aca8cf23ea7074f3 2023-04-18 18:56:28,730 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/07e85023b877485485f207d2cbfce68d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/07e85023b877485485f207d2cbfce68d 2023-04-18 18:56:28,734 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/07e85023b877485485f207d2cbfce68d, entries=150, sequenceid=551, filesize=25.4 K 2023-04-18 18:56:28,735 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/723c28dc9b8e403c9a0f8632b5e908a1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/723c28dc9b8e403c9a0f8632b5e908a1 2023-04-18 18:56:28,749 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/723c28dc9b8e403c9a0f8632b5e908a1, entries=150, sequenceid=551, filesize=11.9 K 2023-04-18 18:56:28,749 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/4a97cdefdc7446e8aca8cf23ea7074f3 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4a97cdefdc7446e8aca8cf23ea7074f3 2023-04-18 18:56:28,753 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4a97cdefdc7446e8aca8cf23ea7074f3, entries=150, sequenceid=551, filesize=11.9 K 2023-04-18 18:56:28,754 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c48561c851d33e29da879cafb393a8fd in 479ms, sequenceid=551, compaction requested=false 2023-04-18 18:56:28,754 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:28,754 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:28,754 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:28,754 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:28,754 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,754 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:28,754 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:28,756 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:28,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:28,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,756 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:28,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:28,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:28,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:28,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:28,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:28,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a9e700a[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:28,758 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:28,758 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,760 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,760 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:28,760 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:28,760 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:28,760 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,761 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:28,762 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,762 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:28,762 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:28,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:28,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:28,762 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:28,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:28,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:28,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:28,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:28,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:28,764 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:28,764 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:28,765 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:28,765 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:28,765 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:28,766 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,766 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:28,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,767 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,767 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:28,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:28,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:28,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:28,768 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:28,771 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:28,771 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:28,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:28,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:28,771 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:28,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:28,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:28,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:28,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:28,771 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:28,771 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:28,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:28,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:28,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:28,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:28,776 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,871 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:28,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:28,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:28,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:28,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:28,875 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:28,875 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:28,875 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:28,875 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:28,876 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,876 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:28,878 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:28,878 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:28,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:28,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:28,879 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,879 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:28,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:28,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:28,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:28,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:28,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:28,880 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:28,880 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:28,880 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:28,880 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:28,880 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:28,880 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:28,880 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:28,881 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-18 18:56:28,881 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:28,881 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:28,881 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:28,881 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:28,881 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:28,881 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:28,894 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418a0114d6386fd44cab2ce80c76f70db4d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a0114d6386fd44cab2ce80c76f70db4d 2023-04-18 18:56:28,899 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=569, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/0f2d4660dc6a46fb9aadc98ac09a866f 2023-04-18 18:56:28,909 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=569 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/c109eb04de884d188ea6c8898e14d9b5 2023-04-18 18:56:28,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:28,982 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:28,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:28,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844248993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:28,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844248995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:28,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:28,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844248995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,038 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/8d3182c560804da8bf54b3a006715d05 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/8d3182c560804da8bf54b3a006715d05 2023-04-18 18:56:29,041 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c48561c851d33e29da879cafb393a8fd/A of c48561c851d33e29da879cafb393a8fd into 8d3182c560804da8bf54b3a006715d05(size=26.6 K), total size for store is 52.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:29,042 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:29,042 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/A, priority=12, startTime=1681844188178; duration=0sec 2023-04-18 18:56:29,042 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:29,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:29,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844249096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:29,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844249098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:29,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844249098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:29,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844249299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,303 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:29,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844249302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:29,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844249302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,318 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=569 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/909cde13d21d4f6093ddbbc161009e3d 2023-04-18 18:56:29,322 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/0f2d4660dc6a46fb9aadc98ac09a866f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/0f2d4660dc6a46fb9aadc98ac09a866f 2023-04-18 18:56:29,326 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/0f2d4660dc6a46fb9aadc98ac09a866f, entries=150, sequenceid=569, filesize=25.4 K 2023-04-18 18:56:29,327 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/c109eb04de884d188ea6c8898e14d9b5 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c109eb04de884d188ea6c8898e14d9b5 2023-04-18 18:56:29,331 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c109eb04de884d188ea6c8898e14d9b5, entries=150, sequenceid=569, filesize=11.9 K 2023-04-18 18:56:29,331 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/909cde13d21d4f6093ddbbc161009e3d as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/909cde13d21d4f6093ddbbc161009e3d 2023-04-18 18:56:29,335 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/909cde13d21d4f6093ddbbc161009e3d, entries=150, sequenceid=569, filesize=11.9 K 2023-04-18 18:56:29,335 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for c48561c851d33e29da879cafb393a8fd in 454ms, sequenceid=569, compaction requested=true 2023-04-18 18:56:29,335 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:29,335 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:29,335 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:29,336 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:29,336 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,336 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:29,336 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:29,338 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,338 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:29,338 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,339 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:29,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:29,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:29,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:29,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6213b1f2[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:29,341 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:29,341 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,342 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,343 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,343 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:29,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,343 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:29,343 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:29,343 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:29,350 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,350 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:29,350 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:29,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:29,350 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:29,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:29,354 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:29,354 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:29,354 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:29,354 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:29,354 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:29,355 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,356 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:29,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,356 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:29,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:29,356 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:29,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:29,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,358 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:29,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,360 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,360 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,361 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,361 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:29,360 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:29,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:29,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:29,361 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:29,361 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:29,361 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:29,361 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,361 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,361 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:29,361 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:29,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:29,461 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:29,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:29,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:29,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:29,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:29,465 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:29,465 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:29,465 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:29,465 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:29,465 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,465 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,467 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:29,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:29,467 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:29,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,468 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,468 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:29,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:29,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:29,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:29,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:29,469 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:29,469 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:29,469 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:29,470 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:29,470 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:29,470 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:29,470 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:29,470 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-18 18:56:29,470 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:29,470 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:29,470 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:29,470 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:29,470 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:29,470 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:29,485 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418e453aae6e7de4b43b4234154036788df to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e453aae6e7de4b43b4234154036788df 2023-04-18 18:56:29,490 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=584, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/de317d44debe4e3c80e2270771b33738 2023-04-18 18:56:29,502 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/b857936d5bdf4237b785ae61f1a541a2 2023-04-18 18:56:29,510 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/46e11776e752486cad90759243cecfa0 2023-04-18 18:56:29,515 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/de317d44debe4e3c80e2270771b33738 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/de317d44debe4e3c80e2270771b33738 2023-04-18 18:56:29,518 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/de317d44debe4e3c80e2270771b33738, entries=150, sequenceid=584, filesize=25.4 K 2023-04-18 18:56:29,519 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/b857936d5bdf4237b785ae61f1a541a2 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b857936d5bdf4237b785ae61f1a541a2 2023-04-18 18:56:29,522 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b857936d5bdf4237b785ae61f1a541a2, entries=150, sequenceid=584, filesize=11.9 K 2023-04-18 18:56:29,523 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/46e11776e752486cad90759243cecfa0 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/46e11776e752486cad90759243cecfa0 2023-04-18 18:56:29,526 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/46e11776e752486cad90759243cecfa0, entries=150, sequenceid=584, filesize=11.9 K 2023-04-18 18:56:29,526 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for c48561c851d33e29da879cafb393a8fd in 56ms, sequenceid=584, compaction requested=true 2023-04-18 18:56:29,526 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:29,526 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:29,526 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:29,527 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:29,527 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,527 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:29,527 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:29,532 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,532 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:29,532 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,532 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:29,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:29,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:29,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:29,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:29,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ad38a25[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:29,534 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:29,534 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,535 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,535 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:29,535 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:29,535 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:29,535 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,536 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:29,537 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:29,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:29,537 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:29,537 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:29,537 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:29,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:29,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:29,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:29,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:29,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:29,540 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:29,540 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:29,540 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:29,540 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:29,540 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:29,542 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,542 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,542 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:29,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:29,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:29,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:29,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:29,542 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,553 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,556 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:29,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:29,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:29,556 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:29,556 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,557 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,557 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,557 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,557 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:29,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:29,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:29,557 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:29,557 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:29,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:29,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:29,557 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,557 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,557 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:29,628 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 18:56:29,628 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:29,628 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:29,628 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:29,628 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:29,628 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:29,628 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:29,650 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041886f73705f2cd415cbfe46085042da2d2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041886f73705f2cd415cbfe46085042da2d2 2023-04-18 18:56:29,657 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:29,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:29,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844189638, max timestamp=1681844189656Num uniques 859; 2023-04-18 18:56:29,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844189638, max timestamp=1681844189656Num uniques 859; 2023-04-18 18:56:29,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:56:29,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681844189638, max timestamp=1681844189656Num uniques 859; 2023-04-18 18:56:29,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:56:29,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 18:56:29,660 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=594, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/ed02bf5303e24a10a8d45cae745a4308 2023-04-18 18:56:29,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:29,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:29,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:29,669 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:29,669 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:29,669 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:29,669 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:29,670 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,670 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,671 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,672 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,672 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:29,673 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/f45a361c6c9e4756b8271f0d5595a91c 2023-04-18 18:56:29,680 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:29,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:29,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:29,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:29,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:29,682 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:29,682 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:29,682 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:29,684 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:29,684 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:29,684 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:29,684 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:29,684 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:29,684 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:29,684 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:29,684 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:29,684 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,684 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:29,684 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:29,686 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,686 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:29,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:29,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:29,687 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,687 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:29,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:29,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:29,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:29,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@513dc679[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:29,688 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:29,688 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,690 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,690 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,690 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:29,690 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:29,690 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:29,690 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:29,692 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:29,692 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,692 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:29,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:29,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:29,693 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:29,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:29,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:29,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844249692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:29,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:29,696 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:29,696 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:29,696 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:29,696 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:29,696 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:29,698 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,698 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:29,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,698 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:29,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:29,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:29,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:29,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844249694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,699 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:29,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844249695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,701 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:29,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,703 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,704 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,704 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,704 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,704 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,704 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:29,704 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:29,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:29,704 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,704 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:29,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:29,704 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:29,704 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:29,704 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:29,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:29,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:29,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844249796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:29,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844249800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:29,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844249801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,804 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:29,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:29,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:29,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:29,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:29,808 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:29,809 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:29,809 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:29,809 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:29,809 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,809 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,811 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:29,811 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:29,811 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,811 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:29,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:29,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:29,812 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:29,813 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:29,813 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:29,813 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:29,813 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:29,813 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:29,813 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:29,813 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:29,813 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:29,813 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:29,813 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:29,813 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,813 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:29,813 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:29,816 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,816 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:29,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:29,816 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,816 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:29,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:29,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:29,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:29,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:29,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@582ae274[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:29,818 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:29,818 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,819 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,819 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,819 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:29,820 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:29,820 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:29,819 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:29,821 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:29,821 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,821 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:29,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:29,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:29,821 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:29,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:29,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:29,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:29,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:29,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:29,825 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:29,825 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:29,825 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:29,825 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:29,825 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:29,826 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,826 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:29,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,826 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:29,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:29,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:29,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:29,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:29,828 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,829 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:29,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,835 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,835 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,835 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,835 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,835 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,835 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,835 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,835 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:29,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:29,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:29,835 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:29,835 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:29,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:29,835 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:29,835 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:29,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:29,936 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:29,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:29,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:29,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:29,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:29,939 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:29,939 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:29,939 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:29,939 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:29,940 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,940 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,943 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,943 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:29,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:29,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:29,944 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,944 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:29,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:29,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:29,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:29,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:29,945 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:29,945 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:29,945 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:29,945 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:29,945 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:29,945 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:29,945 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:29,946 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:29,946 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:29,946 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:29,946 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:29,946 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,946 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:29,946 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:29,947 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,947 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:29,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:29,948 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,948 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:29,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:29,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:29,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:29,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:29,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27c944a2[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:29,949 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:29,949 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,951 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,951 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:29,951 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,951 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:29,951 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:29,951 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:29,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:29,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:29,953 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,953 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:29,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:29,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:29,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:29,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:29,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:29,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:29,956 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:29,956 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:29,956 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:29,956 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:29,956 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:29,958 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,958 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,958 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:29,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:29,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:29,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:29,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:29,958 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,959 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,961 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,962 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:29,962 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,962 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:29,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:29,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:29,962 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:29,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:29,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:29,962 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:29,962 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,962 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:29,962 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:29,962 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:29,962 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:29,962 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:29,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:30,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:30,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844249999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:30,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844250003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:30,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844250003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,062 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:30,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:30,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:30,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:30,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:30,066 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:30,066 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:30,066 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:30,066 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:30,067 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:30,067 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:30,068 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:30,068 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:30,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:30,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:30,069 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,069 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:30,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:30,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:30,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:30,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:30,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:30,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:30,072 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:30,072 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:30,072 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:30,072 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:30,072 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:30,072 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:30,072 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:30,073 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:30,073 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:30,073 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:30,073 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:30,073 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,073 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:30,073 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:30,075 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,075 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:30,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:30,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:30,075 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:30,075 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:30,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:30,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:30,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:30,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:30,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:30,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@618d70dc[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:30,077 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:30,077 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:30,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:30,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:30,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:30,079 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:30,079 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:30,079 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:30,079 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,079 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:30,081 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,081 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:30,081 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:30,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:30,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:30,081 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:30,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:30,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:30,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:30,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:30,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,083 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/e30a6bfb69a14376bb763a465bb32015 2023-04-18 18:56:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:30,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:30,083 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:30,083 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:30,083 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:30,083 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:30,083 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:30,085 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:30,085 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:30,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:30,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:30,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:30,085 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:30,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:30,086 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:30,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:30,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:30,087 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/ed02bf5303e24a10a8d45cae745a4308 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/ed02bf5303e24a10a8d45cae745a4308 2023-04-18 18:56:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:30,087 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:30,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,090 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/ed02bf5303e24a10a8d45cae745a4308, entries=200, sequenceid=594, filesize=32.3 K 2023-04-18 18:56:30,091 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/f45a361c6c9e4756b8271f0d5595a91c as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/f45a361c6c9e4756b8271f0d5595a91c 2023-04-18 18:56:30,093 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,093 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:30,093 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:30,093 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,093 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:30,093 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:30,093 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:30,093 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:30,093 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:30,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:30,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:30,093 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:30,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:30,093 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:30,094 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:30,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:30,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:30,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:30,094 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/f45a361c6c9e4756b8271f0d5595a91c, entries=150, sequenceid=594, filesize=11.9 K 2023-04-18 18:56:30,095 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/e30a6bfb69a14376bb763a465bb32015 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/e30a6bfb69a14376bb763a465bb32015 2023-04-18 18:56:30,098 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/e30a6bfb69a14376bb763a465bb32015, entries=150, sequenceid=594, filesize=11.9 K 2023-04-18 18:56:30,099 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for c48561c851d33e29da879cafb393a8fd in 471ms, sequenceid=594, compaction requested=true 2023-04-18 18:56:30,099 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:30,099 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:30,099 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:56:30,099 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:56:30,099 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:30,099 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:30,100 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 138496 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:56:30,100 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62156 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:56:30,100 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/A is initiating minor compaction (all files) 2023-04-18 18:56:30,100 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/B is initiating minor compaction (all files) 2023-04-18 18:56:30,100 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/A in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:30,100 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/B in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:30,101 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/8d3182c560804da8bf54b3a006715d05, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/07e85023b877485485f207d2cbfce68d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/0f2d4660dc6a46fb9aadc98ac09a866f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/de317d44debe4e3c80e2270771b33738, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/ed02bf5303e24a10a8d45cae745a4308] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=135.3 K 2023-04-18 18:56:30,101 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/72cd48c9169a45bfa30eaeb3392fd66a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/723c28dc9b8e403c9a0f8632b5e908a1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c109eb04de884d188ea6c8898e14d9b5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b857936d5bdf4237b785ae61f1a541a2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/f45a361c6c9e4756b8271f0d5595a91c] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=60.7 K 2023-04-18 18:56:30,101 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 72cd48c9169a45bfa30eaeb3392fd66a, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681844186605 2023-04-18 18:56:30,101 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 8d3182c560804da8bf54b3a006715d05, keycount=150, bloomtype=ROW, size=26.6 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681844186605 2023-04-18 18:56:30,101 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 723c28dc9b8e403c9a0f8632b5e908a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681844187732 2023-04-18 18:56:30,102 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 07e85023b877485485f207d2cbfce68d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681844187732 2023-04-18 18:56:30,102 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting c109eb04de884d188ea6c8898e14d9b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=569, earliestPutTs=1681844188364 2023-04-18 18:56:30,102 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 0f2d4660dc6a46fb9aadc98ac09a866f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=569, earliestPutTs=1681844188364 2023-04-18 18:56:30,102 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting b857936d5bdf4237b785ae61f1a541a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1681844188985 2023-04-18 18:56:30,102 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting de317d44debe4e3c80e2270771b33738, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1681844188985 2023-04-18 18:56:30,102 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting f45a361c6c9e4756b8271f0d5595a91c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681844189613 2023-04-18 18:56:30,103 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting ed02bf5303e24a10a8d45cae745a4308, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681844189613 2023-04-18 18:56:30,124 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#B#compaction#669 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:30,138 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#A#compaction#668 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:30,147 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/5659f05df7a848a8b9125f8651fdc3e6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/5659f05df7a848a8b9125f8651fdc3e6 2023-04-18 18:56:30,153 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c48561c851d33e29da879cafb393a8fd/B of c48561c851d33e29da879cafb393a8fd into 5659f05df7a848a8b9125f8651fdc3e6(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:30,153 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:30,153 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/B, priority=11, startTime=1681844190099; duration=0sec 2023-04-18 18:56:30,153 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:30,153 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 18:56:30,155 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62156 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 18:56:30,155 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/C is initiating minor compaction (all files) 2023-04-18 18:56:30,155 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/C in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:30,155 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/461a005cc02442fcbe813ac9c2ca77f2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4a97cdefdc7446e8aca8cf23ea7074f3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/909cde13d21d4f6093ddbbc161009e3d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/46e11776e752486cad90759243cecfa0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/e30a6bfb69a14376bb763a465bb32015] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=60.7 K 2023-04-18 18:56:30,156 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 461a005cc02442fcbe813ac9c2ca77f2, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681844186605 2023-04-18 18:56:30,156 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 4a97cdefdc7446e8aca8cf23ea7074f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681844187732 2023-04-18 18:56:30,156 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 909cde13d21d4f6093ddbbc161009e3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=569, earliestPutTs=1681844188364 2023-04-18 18:56:30,157 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 46e11776e752486cad90759243cecfa0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1681844188985 2023-04-18 18:56:30,157 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting e30a6bfb69a14376bb763a465bb32015, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681844189613 2023-04-18 18:56:30,163 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#C#compaction#670 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:30,194 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:30,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:30,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:30,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:30,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:30,197 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:30,197 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:30,197 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:30,197 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:30,197 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:30,197 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:30,199 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,199 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:30,199 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:30,199 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:30,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:30,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:30,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:30,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:30,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:30,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:30,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:30,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:30,200 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:30,201 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:30,201 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:30,201 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:30,201 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:30,201 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:30,201 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:30,201 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-18 18:56:30,201 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:30,201 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:30,201 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:30,202 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:30,202 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:30,202 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 18:56:30,216 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304187cfc0d1a6a604d38b9a2f79b61cee114 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304187cfc0d1a6a604d38b9a2f79b61cee114 2023-04-18 18:56:30,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:30,306 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:30,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:30,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844250312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:30,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844250314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:30,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844250315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:30,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844250416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:30,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844250419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:30,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844250419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,577 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/5210de496ead4f06a2c21ea541dcea09 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/5210de496ead4f06a2c21ea541dcea09 2023-04-18 18:56:30,581 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c48561c851d33e29da879cafb393a8fd/C of c48561c851d33e29da879cafb393a8fd into 5210de496ead4f06a2c21ea541dcea09(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:30,581 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:30,581 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/C, priority=11, startTime=1681844190099; duration=0sec 2023-04-18 18:56:30,581 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:30,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:30,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844250620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,622 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=631, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/7ae7af70b38a4066b17d32c6091ea267 2023-04-18 18:56:30,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:30,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844250623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:30,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844250624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:30,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844250922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:30,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:30,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844250927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844250928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:30,966 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/1ab72475f32049b68d2e60bb5bf63946 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/1ab72475f32049b68d2e60bb5bf63946 2023-04-18 18:56:30,971 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c48561c851d33e29da879cafb393a8fd/A of c48561c851d33e29da879cafb393a8fd into 1ab72475f32049b68d2e60bb5bf63946(size=26.8 K), total size for store is 26.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:30,971 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:30,971 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/A, priority=11, startTime=1681844190099; duration=0sec 2023-04-18 18:56:30,971 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:31,038 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/a9777a9da7ab4b9aa6a35cbfbc639aaa 2023-04-18 18:56:31,048 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/1d6e5175276b4b318d3eefb73df35f5f 2023-04-18 18:56:31,052 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/7ae7af70b38a4066b17d32c6091ea267 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/7ae7af70b38a4066b17d32c6091ea267 2023-04-18 18:56:31,055 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/7ae7af70b38a4066b17d32c6091ea267, entries=150, sequenceid=631, filesize=25.4 K 2023-04-18 18:56:31,056 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/a9777a9da7ab4b9aa6a35cbfbc639aaa as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/a9777a9da7ab4b9aa6a35cbfbc639aaa 2023-04-18 18:56:31,059 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/a9777a9da7ab4b9aa6a35cbfbc639aaa, entries=150, sequenceid=631, filesize=11.9 K 2023-04-18 18:56:31,060 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/1d6e5175276b4b318d3eefb73df35f5f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/1d6e5175276b4b318d3eefb73df35f5f 2023-04-18 18:56:31,063 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/1d6e5175276b4b318d3eefb73df35f5f, entries=150, sequenceid=631, filesize=11.9 K 2023-04-18 18:56:31,063 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for c48561c851d33e29da879cafb393a8fd in 862ms, sequenceid=631, compaction requested=false 2023-04-18 18:56:31,064 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:31,064 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:31,064 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:31,064 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:31,064 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,064 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:31,064 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:31,065 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,065 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:31,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:31,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:31,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:31,066 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:31,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:31,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:31,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:31,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:31,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:31,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ebc7c9a[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:31,068 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:31,068 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:31,069 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:31,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:31,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:31,069 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:31,069 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:31,069 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:31,069 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,070 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:31,071 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:31,071 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:31,072 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,072 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:31,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:31,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:31,075 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:31,075 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:31,075 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:31,075 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:31,075 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:31,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:31,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:31,078 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:31,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:31,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:31,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:31,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:31,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:31,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:31,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:31,080 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:31,081 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:31,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:31,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:31,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:31,083 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,083 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:31,083 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:31,083 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:31,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:31,083 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:31,083 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,083 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:31,083 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:31,083 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:31,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:31,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:31,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:31,083 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:31,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:31,084 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:31,084 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:31,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:31,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:31,184 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:31,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:31,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 18:56:31,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 18:56:31,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 18:56:31,187 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 18:56:31,187 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:31,188 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 18:56:31,188 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 18:56:31,188 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:31,188 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:31,191 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:31,191 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:31,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:31,191 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,191 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 18:56:31,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:31,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:31,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 18:56:31,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:31,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 18:56:31,193 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 18:56:31,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 18:56:31,193 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 18:56:31,193 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 18:56:31,193 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 18:56:31,193 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 18:56:31,193 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:31,193 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. started... 2023-04-18 18:56:31,193 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 18:56:31,193 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 18:56:31,194 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:31,194 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:31,194 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:31,194 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:31,194 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:31,194 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:31,214 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418b1d78548b2b04b09a796cb413f8691ca to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418b1d78548b2b04b09a796cb413f8691ca 2023-04-18 18:56:31,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:31,431 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. as already flushing 2023-04-18 18:56:31,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681844191443, max timestamp=1681844191453Num uniques 859; 2023-04-18 18:56:31,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 18:56:31,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844191449, max timestamp=1681844191455Num uniques 859; 2023-04-18 18:56:31,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:56:31,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681844191449, max timestamp=1681844191455Num uniques 859; 2023-04-18 18:56:31,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 18:56:31,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:31,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844251476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:31,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844251476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,480 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:31,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844251477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:31,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844251579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:31,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34053] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844251579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:31,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844251582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,620 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=643, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/395f2b0e38484c0084ab238d38efea06 2023-04-18 18:56:31,630 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=643 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/1a98872522164e7fb93865edfd87aee9 2023-04-18 18:56:31,781 DEBUG [Thread-3992] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5fdfef3b to 127.0.0.1:65091 2023-04-18 18:56:31,781 DEBUG [Thread-3990] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x785e366b to 127.0.0.1:65091 2023-04-18 18:56:31,782 DEBUG [Thread-3992] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:31,782 DEBUG [Thread-3990] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:31,782 DEBUG [Thread-3986] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a709b21 to 127.0.0.1:65091 2023-04-18 18:56:31,782 DEBUG [Thread-3986] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:31,783 DEBUG [Thread-3988] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x156e4ba6 to 127.0.0.1:65091 2023-04-18 18:56:31,783 DEBUG [Thread-3988] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:31,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:31,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:31,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34053] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43936 deadline: 1681844251783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43886 deadline: 1681844251784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:31,784 DEBUG [Thread-3984] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c1a68cd to 127.0.0.1:65091 2023-04-18 18:56:31,784 DEBUG [Thread-3984] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:31,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 18:56:31,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43916 deadline: 1681844251786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:32,040 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=643 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/9410dcc62ac74ac18c25ad8d5e4a0ba6 2023-04-18 18:56:32,044 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/395f2b0e38484c0084ab238d38efea06 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/395f2b0e38484c0084ab238d38efea06 2023-04-18 18:56:32,046 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/395f2b0e38484c0084ab238d38efea06, entries=100, sequenceid=643, filesize=18.6 K 2023-04-18 18:56:32,046 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/1a98872522164e7fb93865edfd87aee9 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/1a98872522164e7fb93865edfd87aee9 2023-04-18 18:56:32,049 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/1a98872522164e7fb93865edfd87aee9, entries=100, sequenceid=643, filesize=9.5 K 2023-04-18 18:56:32,049 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/9410dcc62ac74ac18c25ad8d5e4a0ba6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9410dcc62ac74ac18c25ad8d5e4a0ba6 2023-04-18 18:56:32,052 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9410dcc62ac74ac18c25ad8d5e4a0ba6, entries=100, sequenceid=643, filesize=9.5 K 2023-04-18 18:56:32,052 INFO [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.94 KB/135110 for c48561c851d33e29da879cafb393a8fd in 859ms, sequenceid=643, compaction requested=true 2023-04-18 18:56:32,052 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:32,052 DEBUG [rs(jenkins-hbase4.apache.org,34053,1681844018952)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:32,053 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 18:56:32,053 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 18:56:32,053 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:32,053 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 18:56:32,053 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 18:56:32,054 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:32,054 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:32,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:32,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:32,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:32,054 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:32,054 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 18:56:32,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:32,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:32,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:32,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:32,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:32,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34053,1681844018952' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 18:56:32,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@510a0c7b[Count = 0] remaining members to acquire global barrier 2023-04-18 18:56:32,055 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 18:56:32,055 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:32,056 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:32,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:32,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:32,056 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 18:56:32,056 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 18:56:32,056 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34053,1681844018952' in zk 2023-04-18 18:56:32,056 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:32,057 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 18:56:32,058 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 18:56:32,058 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:32,058 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:32,058 DEBUG [member: 'jenkins-hbase4.apache.org,34053,1681844018952' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 18:56:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:32,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:32,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:32,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:32,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34053,1681844018952': 2023-04-18 18:56:32,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34053,1681844018952' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 18:56:32,059 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 18:56:32,059 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 18:56:32,059 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 18:56:32,059 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 18:56:32,060 INFO [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 18:56:32,062 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:32,062 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:32,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:32,062 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 18:56:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 18:56:32,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:32,063 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:32,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 18:56:32,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:32,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:32,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 18:56:32,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:32,064 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 18:56:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 18:56:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:32,065 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:32,065 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 18:56:32,065 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:32,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 18:56:32,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 18:56:32,065 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 18:56:32,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 18:56:32,066 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:32,065 DEBUG [(jenkins-hbase4.apache.org,39391,1681844017730)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 18:56:32,065 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 18:56:32,066 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:32,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 18:56:32,066 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 18:56:32,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 18:56:32,066 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 18:56:32,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 18:56:32,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 18:56:32,066 DEBUG [Thread-3983] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 18:56:32,066 DEBUG [Thread-3983] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 18:56:32,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34053] regionserver.HRegion(9158): Flush requested on c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:32,088 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=145.36 KB heapSize=415.77 KB 2023-04-18 18:56:32,088 DEBUG [Thread-3981] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x526f2221 to 127.0.0.1:65091 2023-04-18 18:56:32,088 DEBUG [Thread-3975] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52104714 to 127.0.0.1:65091 2023-04-18 18:56:32,088 DEBUG [Thread-3981] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:32,088 DEBUG [Thread-3975] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:32,088 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:32,089 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 18:56:32,089 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:32,089 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 18:56:32,089 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:32,089 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 18:56:32,090 DEBUG [Thread-3977] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4a320817 to 127.0.0.1:65091 2023-04-18 18:56:32,090 DEBUG [Thread-3977] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:32,099 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304189a85c9238ad149b88029e2dbda3df9b4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304189a85c9238ad149b88029e2dbda3df9b4 2023-04-18 18:56:32,166 DEBUG [Thread-3983] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 18:56:32,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 18:56:32,503 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=681, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/55bf159efb9141bc8c9621006308375a 2023-04-18 18:56:32,511 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=681 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/ffd94c4f1b1946c8837f44b1b535be59 2023-04-18 18:56:32,918 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=681 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/9183006fbfa446e48b16c2ac8e8475bb 2023-04-18 18:56:32,921 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/55bf159efb9141bc8c9621006308375a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/55bf159efb9141bc8c9621006308375a 2023-04-18 18:56:32,924 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/55bf159efb9141bc8c9621006308375a, entries=150, sequenceid=681, filesize=25.4 K 2023-04-18 18:56:32,924 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/ffd94c4f1b1946c8837f44b1b535be59 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/ffd94c4f1b1946c8837f44b1b535be59 2023-04-18 18:56:32,927 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/ffd94c4f1b1946c8837f44b1b535be59, entries=150, sequenceid=681, filesize=11.9 K 2023-04-18 18:56:32,928 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/9183006fbfa446e48b16c2ac8e8475bb as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9183006fbfa446e48b16c2ac8e8475bb 2023-04-18 18:56:32,930 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9183006fbfa446e48b16c2ac8e8475bb, entries=150, sequenceid=681, filesize=11.9 K 2023-04-18 18:56:32,931 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~145.36 KB/148850, heapSize ~415.72 KB/425696, currentSize=6.71 KB/6870 for c48561c851d33e29da879cafb393a8fd in 843ms, sequenceid=681, compaction requested=true 2023-04-18 18:56:32,931 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:32,931 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:32,931 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:32,931 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:56:32,931 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:56:32,931 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:32,932 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 98532 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:56:32,932 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47696 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:56:32,932 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/A is initiating minor compaction (all files) 2023-04-18 18:56:32,932 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/B is initiating minor compaction (all files) 2023-04-18 18:56:32,932 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/B in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:32,932 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/A in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:32,932 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/5659f05df7a848a8b9125f8651fdc3e6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/a9777a9da7ab4b9aa6a35cbfbc639aaa, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/1a98872522164e7fb93865edfd87aee9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/ffd94c4f1b1946c8837f44b1b535be59] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=46.6 K 2023-04-18 18:56:32,932 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/1ab72475f32049b68d2e60bb5bf63946, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/7ae7af70b38a4066b17d32c6091ea267, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/395f2b0e38484c0084ab238d38efea06, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/55bf159efb9141bc8c9621006308375a] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=96.2 K 2023-04-18 18:56:32,933 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 5659f05df7a848a8b9125f8651fdc3e6, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681844189613 2023-04-18 18:56:32,933 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 1ab72475f32049b68d2e60bb5bf63946, keycount=150, bloomtype=ROW, size=26.8 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681844189613 2023-04-18 18:56:32,933 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting a9777a9da7ab4b9aa6a35cbfbc639aaa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=631, earliestPutTs=1681844189675 2023-04-18 18:56:32,933 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 7ae7af70b38a4066b17d32c6091ea267, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=631, earliestPutTs=1681844189675 2023-04-18 18:56:32,933 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 1a98872522164e7fb93865edfd87aee9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=643, earliestPutTs=1681844190309 2023-04-18 18:56:32,933 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 395f2b0e38484c0084ab238d38efea06, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=643, earliestPutTs=1681844190309 2023-04-18 18:56:32,933 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting ffd94c4f1b1946c8837f44b1b535be59, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=681, earliestPutTs=1681844191474 2023-04-18 18:56:32,933 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] compactions.Compactor(207): Compacting 55bf159efb9141bc8c9621006308375a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=681, earliestPutTs=1681844191474 2023-04-18 18:56:32,949 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#B#compaction#681 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:32,950 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#A#compaction#680 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:33,360 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/33cdcd167ae24ffe969b4f60b21744ae as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/33cdcd167ae24ffe969b4f60b21744ae 2023-04-18 18:56:33,363 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c48561c851d33e29da879cafb393a8fd/B of c48561c851d33e29da879cafb393a8fd into 33cdcd167ae24ffe969b4f60b21744ae(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:33,363 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:33,363 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/B, priority=12, startTime=1681844192931; duration=0sec 2023-04-18 18:56:33,364 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 18:56:33,364 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 18:56:33,364 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47696 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 18:56:33,365 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1912): c48561c851d33e29da879cafb393a8fd/C is initiating minor compaction (all files) 2023-04-18 18:56:33,365 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c48561c851d33e29da879cafb393a8fd/C in TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:33,365 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/5210de496ead4f06a2c21ea541dcea09, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/1d6e5175276b4b318d3eefb73df35f5f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9410dcc62ac74ac18c25ad8d5e4a0ba6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9183006fbfa446e48b16c2ac8e8475bb] into tmpdir=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp, totalSize=46.6 K 2023-04-18 18:56:33,365 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 5210de496ead4f06a2c21ea541dcea09, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681844189613 2023-04-18 18:56:33,365 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 1d6e5175276b4b318d3eefb73df35f5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=631, earliestPutTs=1681844189675 2023-04-18 18:56:33,365 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 9410dcc62ac74ac18c25ad8d5e4a0ba6, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=643, earliestPutTs=1681844190309 2023-04-18 18:56:33,366 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] compactions.Compactor(207): Compacting 9183006fbfa446e48b16c2ac8e8475bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=681, earliestPutTs=1681844191474 2023-04-18 18:56:33,371 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] throttle.PressureAwareThroughputController(145): c48561c851d33e29da879cafb393a8fd#C#compaction#682 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 18:56:33,377 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/0307b89f1f254ba3a4d56e570e941561 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/0307b89f1f254ba3a4d56e570e941561 2023-04-18 18:56:33,381 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c48561c851d33e29da879cafb393a8fd/C of c48561c851d33e29da879cafb393a8fd into 0307b89f1f254ba3a4d56e570e941561(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 18:56:33,381 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:33,381 INFO [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/C, priority=12, startTime=1681844192931; duration=0sec 2023-04-18 18:56:33,381 DEBUG [RS:0;jenkins-hbase4:34053-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:34,165 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/ab3f684803094a5a9a339fe1978ce44f as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/ab3f684803094a5a9a339fe1978ce44f 2023-04-18 18:56:34,168 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c48561c851d33e29da879cafb393a8fd/A of c48561c851d33e29da879cafb393a8fd into ab3f684803094a5a9a339fe1978ce44f(size=26.9 K), total size for store is 26.9 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-18 18:56:34,168 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:34,168 INFO [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd., storeName=c48561c851d33e29da879cafb393a8fd/A, priority=12, startTime=1681844192931; duration=1sec 2023-04-18 18:56:34,168 DEBUG [RS:0;jenkins-hbase4:34053-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 18:56:36,168 DEBUG [Thread-3979] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x19c1e3ee to 127.0.0.1:65091 2023-04-18 18:56:36,168 DEBUG [Thread-3979] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:36,182 DEBUG [Thread-3973] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6257294f to 127.0.0.1:65091 2023-04-18 18:56:36,182 DEBUG [Thread-3973] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:36,182 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-18 18:56:36,182 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 49 2023-04-18 18:56:36,182 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 164 2023-04-18 18:56:36,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 164 2023-04-18 18:56:36,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 53 2023-04-18 18:56:36,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(394): wrote 112 2023-04-18 18:56:36,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-18 18:56:36,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(398): read 4458 2023-04-18 18:56:36,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(398): read 4454 2023-04-18 18:56:36,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(398): read 4399 2023-04-18 18:56:36,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(398): read 4479 2023-04-18 18:56:36,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(398): read 4442 2023-04-18 18:56:36,183 INFO [Listener at localhost/36951] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-18 18:56:36,183 INFO [Listener at localhost/36951] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-18 18:56:36,183 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0c0766f0 to 127.0.0.1:65091 2023-04-18 18:56:36,183 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:36,184 INFO [Listener at localhost/36951] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-18 18:56:36,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-18 18:56:36,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-18 18:56:36,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-18 18:56:36,187 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844196187"}]},"ts":"1681844196187"} 2023-04-18 18:56:36,187 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-18 18:56:36,189 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-18 18:56:36,189 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c48561c851d33e29da879cafb393a8fd, UNASSIGN}] 2023-04-18 18:56:36,191 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c48561c851d33e29da879cafb393a8fd, UNASSIGN 2023-04-18 18:56:36,191 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=c48561c851d33e29da879cafb393a8fd, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:36,191 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844196191"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681844196191"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681844196191"}]},"ts":"1681844196191"} 2023-04-18 18:56:36,192 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952}] 2023-04-18 18:56:36,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-18 18:56:36,343 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:36,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c48561c851d33e29da879cafb393a8fd, disabling compactions & flushes 2023-04-18 18:56:36,343 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:36,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:36,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. after waiting 0 ms 2023-04-18 18:56:36,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:36,344 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing c48561c851d33e29da879cafb393a8fd 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-18 18:56:36,344 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=A 2023-04-18 18:56:36,344 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:36,344 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=B 2023-04-18 18:56:36,344 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:36,344 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c48561c851d33e29da879cafb393a8fd, store=C 2023-04-18 18:56:36,344 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 18:56:36,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-18 18:56:36,753 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304185e1df17d591c4300a9a270e7b6d54511 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304185e1df17d591c4300a9a270e7b6d54511 2023-04-18 18:56:36,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-18 18:56:37,158 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=690, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/6867aa88cc6a4ef1a2e1e86f715fea49 2023-04-18 18:56:37,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-18 18:56:37,566 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=690 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/05a2bb20fb8f4e2fb0e5b4742f8ff4a6 2023-04-18 18:56:37,573 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=690 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/8d87d99d82c6451c8fc77693aa2e60cb 2023-04-18 18:56:37,576 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/A/6867aa88cc6a4ef1a2e1e86f715fea49 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/6867aa88cc6a4ef1a2e1e86f715fea49 2023-04-18 18:56:37,579 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/6867aa88cc6a4ef1a2e1e86f715fea49, entries=100, sequenceid=690, filesize=18.6 K 2023-04-18 18:56:37,579 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/B/05a2bb20fb8f4e2fb0e5b4742f8ff4a6 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/05a2bb20fb8f4e2fb0e5b4742f8ff4a6 2023-04-18 18:56:37,582 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/05a2bb20fb8f4e2fb0e5b4742f8ff4a6, entries=100, sequenceid=690, filesize=9.5 K 2023-04-18 18:56:37,582 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/.tmp/C/8d87d99d82c6451c8fc77693aa2e60cb as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/8d87d99d82c6451c8fc77693aa2e60cb 2023-04-18 18:56:37,585 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/8d87d99d82c6451c8fc77693aa2e60cb, entries=100, sequenceid=690, filesize=9.5 K 2023-04-18 18:56:37,585 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for c48561c851d33e29da879cafb393a8fd in 1241ms, sequenceid=690, compaction requested=false 2023-04-18 18:56:37,609 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/9e367509f9094d9d989c77c04f1f4829, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/f9153095797c430fab62ca852f6ad1f7, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/6d0233c06fd544ad8d9d6eb1a5095b10, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/0ca335a3162e4c738b1372c1da1aa001, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/f8e10ad267914a1b94cd552c9e7668c8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/c4336d4de34841ec924faf183203cd0d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/586084c7f8454b808b2ca800c3547616, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/ad3dc098af214a6da75afd4ff3caa85b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/974e5156e04b44d092854b05711126e4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/89f97c0dc85c4c4195e82a1b9b141337, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/bb4b45fa96664dadaa72ffb577b9bd40, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/2fc3aee7c71149d9b02450907900d77d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/4003c9cc3cee44d9b18e213c94152fc4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/19ba411a86cc4edebb0e655a46de6479, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/9afd9b7966e44bb8bcf6832e9cc79aa8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/d7ed443a5344422c8ebe9a9515cb81c4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/9a78b8d44a9540dfa3e0f78aedf13bf2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/a2ee948d72e743269fef7f009a89cadf, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/99088948e48a46f3a41a48d91cbcc21d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/4f5139d197f242289c581cc65378a915, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/04de38b715764571a68e3f36f6bb80d6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/7aa2256b090f453e9284ef2439654a01, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/bef46d0ed3774f0bb0b799ce8907a3dc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/70f1a5f4472f42d2b9d0d74af634497d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/1eefdae08ee146228dfd333297324f29, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/e04f88ce51b14d45b5f879efd88d1445, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/0a93a96a16464f3cb20179daa3ce95d7, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/3dc0f98887064e4cb31100d4ce47c7b9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/212e3d596b6a4111b3e0a257ab8ccde6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/4d1c48c367444ea4bc66ccf05825e9f2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/47857ad268b64f15a8779a02b7fd669f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/169bbddb8445404b985204c4c2c41f1f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/ff40a39f7a994c4ab96c668d3c6950ac, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/2779ebd33a3f4b44ad9551ab5d50eeb6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/7b0c1cd7b76d479d84b93997422e4c2c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/35d1cdfee9344e0bb9b82a603fe41410, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/8d3182c560804da8bf54b3a006715d05, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/07e85023b877485485f207d2cbfce68d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/0f2d4660dc6a46fb9aadc98ac09a866f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/de317d44debe4e3c80e2270771b33738, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/ed02bf5303e24a10a8d45cae745a4308, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/1ab72475f32049b68d2e60bb5bf63946, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/7ae7af70b38a4066b17d32c6091ea267, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/395f2b0e38484c0084ab238d38efea06, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/55bf159efb9141bc8c9621006308375a] to archive 2023-04-18 18:56:37,610 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 18:56:37,611 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/9e367509f9094d9d989c77c04f1f4829 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/9e367509f9094d9d989c77c04f1f4829 2023-04-18 18:56:37,612 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/f9153095797c430fab62ca852f6ad1f7 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/f9153095797c430fab62ca852f6ad1f7 2023-04-18 18:56:37,613 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/6d0233c06fd544ad8d9d6eb1a5095b10 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/6d0233c06fd544ad8d9d6eb1a5095b10 2023-04-18 18:56:37,614 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/0ca335a3162e4c738b1372c1da1aa001 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/0ca335a3162e4c738b1372c1da1aa001 2023-04-18 18:56:37,615 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/f8e10ad267914a1b94cd552c9e7668c8 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/f8e10ad267914a1b94cd552c9e7668c8 2023-04-18 18:56:37,615 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/c4336d4de34841ec924faf183203cd0d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/c4336d4de34841ec924faf183203cd0d 2023-04-18 18:56:37,616 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/586084c7f8454b808b2ca800c3547616 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/586084c7f8454b808b2ca800c3547616 2023-04-18 18:56:37,617 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/ad3dc098af214a6da75afd4ff3caa85b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/ad3dc098af214a6da75afd4ff3caa85b 2023-04-18 18:56:37,618 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/974e5156e04b44d092854b05711126e4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/974e5156e04b44d092854b05711126e4 2023-04-18 18:56:37,619 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/89f97c0dc85c4c4195e82a1b9b141337 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/89f97c0dc85c4c4195e82a1b9b141337 2023-04-18 18:56:37,619 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/bb4b45fa96664dadaa72ffb577b9bd40 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/bb4b45fa96664dadaa72ffb577b9bd40 2023-04-18 18:56:37,620 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/2fc3aee7c71149d9b02450907900d77d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/2fc3aee7c71149d9b02450907900d77d 2023-04-18 18:56:37,621 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/4003c9cc3cee44d9b18e213c94152fc4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/4003c9cc3cee44d9b18e213c94152fc4 2023-04-18 18:56:37,622 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/19ba411a86cc4edebb0e655a46de6479 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/19ba411a86cc4edebb0e655a46de6479 2023-04-18 18:56:37,622 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/9afd9b7966e44bb8bcf6832e9cc79aa8 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/9afd9b7966e44bb8bcf6832e9cc79aa8 2023-04-18 18:56:37,623 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/d7ed443a5344422c8ebe9a9515cb81c4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/d7ed443a5344422c8ebe9a9515cb81c4 2023-04-18 18:56:37,624 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/9a78b8d44a9540dfa3e0f78aedf13bf2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/9a78b8d44a9540dfa3e0f78aedf13bf2 2023-04-18 18:56:37,625 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/a2ee948d72e743269fef7f009a89cadf to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/a2ee948d72e743269fef7f009a89cadf 2023-04-18 18:56:37,626 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/99088948e48a46f3a41a48d91cbcc21d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/99088948e48a46f3a41a48d91cbcc21d 2023-04-18 18:56:37,626 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/4f5139d197f242289c581cc65378a915 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/4f5139d197f242289c581cc65378a915 2023-04-18 18:56:37,627 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/04de38b715764571a68e3f36f6bb80d6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/04de38b715764571a68e3f36f6bb80d6 2023-04-18 18:56:37,628 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/7aa2256b090f453e9284ef2439654a01 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/7aa2256b090f453e9284ef2439654a01 2023-04-18 18:56:37,629 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/bef46d0ed3774f0bb0b799ce8907a3dc to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/bef46d0ed3774f0bb0b799ce8907a3dc 2023-04-18 18:56:37,630 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/70f1a5f4472f42d2b9d0d74af634497d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/70f1a5f4472f42d2b9d0d74af634497d 2023-04-18 18:56:37,630 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/1eefdae08ee146228dfd333297324f29 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/1eefdae08ee146228dfd333297324f29 2023-04-18 18:56:37,631 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/e04f88ce51b14d45b5f879efd88d1445 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/e04f88ce51b14d45b5f879efd88d1445 2023-04-18 18:56:37,632 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/0a93a96a16464f3cb20179daa3ce95d7 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/0a93a96a16464f3cb20179daa3ce95d7 2023-04-18 18:56:37,633 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/3dc0f98887064e4cb31100d4ce47c7b9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/3dc0f98887064e4cb31100d4ce47c7b9 2023-04-18 18:56:37,634 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/212e3d596b6a4111b3e0a257ab8ccde6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/212e3d596b6a4111b3e0a257ab8ccde6 2023-04-18 18:56:37,635 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/4d1c48c367444ea4bc66ccf05825e9f2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/4d1c48c367444ea4bc66ccf05825e9f2 2023-04-18 18:56:37,635 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/47857ad268b64f15a8779a02b7fd669f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/47857ad268b64f15a8779a02b7fd669f 2023-04-18 18:56:37,636 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/169bbddb8445404b985204c4c2c41f1f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/169bbddb8445404b985204c4c2c41f1f 2023-04-18 18:56:37,637 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/ff40a39f7a994c4ab96c668d3c6950ac to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/ff40a39f7a994c4ab96c668d3c6950ac 2023-04-18 18:56:37,638 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/2779ebd33a3f4b44ad9551ab5d50eeb6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/2779ebd33a3f4b44ad9551ab5d50eeb6 2023-04-18 18:56:37,639 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/7b0c1cd7b76d479d84b93997422e4c2c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/7b0c1cd7b76d479d84b93997422e4c2c 2023-04-18 18:56:37,640 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/35d1cdfee9344e0bb9b82a603fe41410 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/35d1cdfee9344e0bb9b82a603fe41410 2023-04-18 18:56:37,640 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/8d3182c560804da8bf54b3a006715d05 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/8d3182c560804da8bf54b3a006715d05 2023-04-18 18:56:37,641 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/07e85023b877485485f207d2cbfce68d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/07e85023b877485485f207d2cbfce68d 2023-04-18 18:56:37,642 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/0f2d4660dc6a46fb9aadc98ac09a866f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/0f2d4660dc6a46fb9aadc98ac09a866f 2023-04-18 18:56:37,643 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/de317d44debe4e3c80e2270771b33738 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/de317d44debe4e3c80e2270771b33738 2023-04-18 18:56:37,644 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/ed02bf5303e24a10a8d45cae745a4308 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/ed02bf5303e24a10a8d45cae745a4308 2023-04-18 18:56:37,645 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/1ab72475f32049b68d2e60bb5bf63946 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/1ab72475f32049b68d2e60bb5bf63946 2023-04-18 18:56:37,645 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/7ae7af70b38a4066b17d32c6091ea267 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/7ae7af70b38a4066b17d32c6091ea267 2023-04-18 18:56:37,646 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/395f2b0e38484c0084ab238d38efea06 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/395f2b0e38484c0084ab238d38efea06 2023-04-18 18:56:37,647 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/55bf159efb9141bc8c9621006308375a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/55bf159efb9141bc8c9621006308375a 2023-04-18 18:56:37,651 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/0eff99bcdbd746f687208d0a8061f09e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/59d6a9882f194717872cc7827af5e18b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/71c05c954d8346dfa0f12267381e8952, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/a2c42639e660462ea1cdf242afc36547, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/9c1ea6a8d6194fe0a06d94582cf52918, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/83ef252fdc094e348afc378dafb415f0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b849c2de926940709d04ca702c728628, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/cfbcd3f910dd4d9e8bdff8797dd53044, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b970553f344c4953b30a213313232a6c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fa7b440a161946efb195b6e2cf07dbb3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/3302e82a60964e80a8ac241c16c32f2f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/4d124fd5e4ce413987828181719d8a05, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/349845958e134ee68ba9fb0e364f386d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fcdfa72ef109489eb1c733748c86f3a1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c3f596b1844e42d793fe4d808524caa5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/850146392e4b4b29895c506e5bd75fd4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/d33d9be279f041319ac29246a5bfe1ae, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c89b87f3f23d4be4a2fa9cee24de5681, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/5016fed21fb64ded8b0356007c813113, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/3c438fad9a404731a972ba8893ca5eb8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/97daa835736f4183bb7e293500726ee1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/a187514ec0714421845a383f566396da, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/256116a4659d4790bc566ba2c8e6b563, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c720c7c773d741159ac127fdb022d737, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/810eeec11f414b138733f9dc5e5ab56d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b6fb5123883844a09e308ac6506c27b4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/690a9ba04de543edaa9b0ba151f376e1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/13bd1548f83f4b698345260613a0d29c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/9944eb7c7382437194b8384133328f7e, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/d085ea0e640f4e25943a48821501665f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/7831a5574d7448818a75dfc827259c7b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/bd4ce00e084247c1b4b6c9b66f05ac3b, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b74aa7b83cc043f19dceba3a360837d0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fafad7032e624390aa39cc48ff46e8c8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fc3bc2f048d94a8cb9ed0b9b1deed10d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/72cd48c9169a45bfa30eaeb3392fd66a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/8d4842bc2eba4954a890ddc4d9fb5076, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/723c28dc9b8e403c9a0f8632b5e908a1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c109eb04de884d188ea6c8898e14d9b5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b857936d5bdf4237b785ae61f1a541a2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/5659f05df7a848a8b9125f8651fdc3e6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/f45a361c6c9e4756b8271f0d5595a91c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/a9777a9da7ab4b9aa6a35cbfbc639aaa, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/1a98872522164e7fb93865edfd87aee9, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/ffd94c4f1b1946c8837f44b1b535be59] to archive 2023-04-18 18:56:37,651 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 18:56:37,653 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/0eff99bcdbd746f687208d0a8061f09e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/0eff99bcdbd746f687208d0a8061f09e 2023-04-18 18:56:37,654 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/59d6a9882f194717872cc7827af5e18b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/59d6a9882f194717872cc7827af5e18b 2023-04-18 18:56:37,656 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/71c05c954d8346dfa0f12267381e8952 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/71c05c954d8346dfa0f12267381e8952 2023-04-18 18:56:37,657 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/a2c42639e660462ea1cdf242afc36547 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/a2c42639e660462ea1cdf242afc36547 2023-04-18 18:56:37,658 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/9c1ea6a8d6194fe0a06d94582cf52918 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/9c1ea6a8d6194fe0a06d94582cf52918 2023-04-18 18:56:37,659 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/83ef252fdc094e348afc378dafb415f0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/83ef252fdc094e348afc378dafb415f0 2023-04-18 18:56:37,661 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b849c2de926940709d04ca702c728628 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b849c2de926940709d04ca702c728628 2023-04-18 18:56:37,662 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/cfbcd3f910dd4d9e8bdff8797dd53044 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/cfbcd3f910dd4d9e8bdff8797dd53044 2023-04-18 18:56:37,663 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b970553f344c4953b30a213313232a6c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b970553f344c4953b30a213313232a6c 2023-04-18 18:56:37,664 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fa7b440a161946efb195b6e2cf07dbb3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fa7b440a161946efb195b6e2cf07dbb3 2023-04-18 18:56:37,666 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/3302e82a60964e80a8ac241c16c32f2f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/3302e82a60964e80a8ac241c16c32f2f 2023-04-18 18:56:37,667 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/4d124fd5e4ce413987828181719d8a05 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/4d124fd5e4ce413987828181719d8a05 2023-04-18 18:56:37,668 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/349845958e134ee68ba9fb0e364f386d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/349845958e134ee68ba9fb0e364f386d 2023-04-18 18:56:37,669 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fcdfa72ef109489eb1c733748c86f3a1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fcdfa72ef109489eb1c733748c86f3a1 2023-04-18 18:56:37,670 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c3f596b1844e42d793fe4d808524caa5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c3f596b1844e42d793fe4d808524caa5 2023-04-18 18:56:37,672 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/850146392e4b4b29895c506e5bd75fd4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/850146392e4b4b29895c506e5bd75fd4 2023-04-18 18:56:37,673 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/d33d9be279f041319ac29246a5bfe1ae to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/d33d9be279f041319ac29246a5bfe1ae 2023-04-18 18:56:37,674 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c89b87f3f23d4be4a2fa9cee24de5681 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c89b87f3f23d4be4a2fa9cee24de5681 2023-04-18 18:56:37,675 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/5016fed21fb64ded8b0356007c813113 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/5016fed21fb64ded8b0356007c813113 2023-04-18 18:56:37,676 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/3c438fad9a404731a972ba8893ca5eb8 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/3c438fad9a404731a972ba8893ca5eb8 2023-04-18 18:56:37,677 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/97daa835736f4183bb7e293500726ee1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/97daa835736f4183bb7e293500726ee1 2023-04-18 18:56:37,678 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/a187514ec0714421845a383f566396da to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/a187514ec0714421845a383f566396da 2023-04-18 18:56:37,679 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/256116a4659d4790bc566ba2c8e6b563 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/256116a4659d4790bc566ba2c8e6b563 2023-04-18 18:56:37,680 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c720c7c773d741159ac127fdb022d737 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c720c7c773d741159ac127fdb022d737 2023-04-18 18:56:37,681 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/810eeec11f414b138733f9dc5e5ab56d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/810eeec11f414b138733f9dc5e5ab56d 2023-04-18 18:56:37,682 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b6fb5123883844a09e308ac6506c27b4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b6fb5123883844a09e308ac6506c27b4 2023-04-18 18:56:37,683 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/690a9ba04de543edaa9b0ba151f376e1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/690a9ba04de543edaa9b0ba151f376e1 2023-04-18 18:56:37,684 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/13bd1548f83f4b698345260613a0d29c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/13bd1548f83f4b698345260613a0d29c 2023-04-18 18:56:37,685 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/9944eb7c7382437194b8384133328f7e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/9944eb7c7382437194b8384133328f7e 2023-04-18 18:56:37,686 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/d085ea0e640f4e25943a48821501665f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/d085ea0e640f4e25943a48821501665f 2023-04-18 18:56:37,686 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/7831a5574d7448818a75dfc827259c7b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/7831a5574d7448818a75dfc827259c7b 2023-04-18 18:56:37,687 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/bd4ce00e084247c1b4b6c9b66f05ac3b to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/bd4ce00e084247c1b4b6c9b66f05ac3b 2023-04-18 18:56:37,688 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b74aa7b83cc043f19dceba3a360837d0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b74aa7b83cc043f19dceba3a360837d0 2023-04-18 18:56:37,689 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fafad7032e624390aa39cc48ff46e8c8 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fafad7032e624390aa39cc48ff46e8c8 2023-04-18 18:56:37,690 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fc3bc2f048d94a8cb9ed0b9b1deed10d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/fc3bc2f048d94a8cb9ed0b9b1deed10d 2023-04-18 18:56:37,691 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/72cd48c9169a45bfa30eaeb3392fd66a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/72cd48c9169a45bfa30eaeb3392fd66a 2023-04-18 18:56:37,692 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/8d4842bc2eba4954a890ddc4d9fb5076 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/8d4842bc2eba4954a890ddc4d9fb5076 2023-04-18 18:56:37,693 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/723c28dc9b8e403c9a0f8632b5e908a1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/723c28dc9b8e403c9a0f8632b5e908a1 2023-04-18 18:56:37,694 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c109eb04de884d188ea6c8898e14d9b5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/c109eb04de884d188ea6c8898e14d9b5 2023-04-18 18:56:37,695 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b857936d5bdf4237b785ae61f1a541a2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/b857936d5bdf4237b785ae61f1a541a2 2023-04-18 18:56:37,696 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/5659f05df7a848a8b9125f8651fdc3e6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/5659f05df7a848a8b9125f8651fdc3e6 2023-04-18 18:56:37,698 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/f45a361c6c9e4756b8271f0d5595a91c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/f45a361c6c9e4756b8271f0d5595a91c 2023-04-18 18:56:37,699 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/a9777a9da7ab4b9aa6a35cbfbc639aaa to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/a9777a9da7ab4b9aa6a35cbfbc639aaa 2023-04-18 18:56:37,700 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/1a98872522164e7fb93865edfd87aee9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/1a98872522164e7fb93865edfd87aee9 2023-04-18 18:56:37,701 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/ffd94c4f1b1946c8837f44b1b535be59 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/ffd94c4f1b1946c8837f44b1b535be59 2023-04-18 18:56:37,705 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9c17dc07c85c48bf82ec6b1816214534, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/c592bb64f5fc420aa49be513f17e4ef5, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/5d1115953ab64140a00444ef9d0f7d68, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4b03ad08bfeb43788d8d757c39d178f1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9c330e9ad72b43648a8c79ddbb89f661, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/24d197483e164a028e5c59c75d1b287c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/65f50c4bf82149c7a23defc4fcc3b7c4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4079798575dd46b08e8e44e691d8b6ef, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/f14ee3a998be4ca89f736e1fa6adde27, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/241323bdc5704938bc1c0bda5b86db8a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/b2d9d644ae7e4b6a8820dc348d1c173a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/78a7ebbe6df749199cf84f04c9cccac3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/527cdd887eb4479e90d8243bda973530, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/70804a300c8243768d488a85bfc653f1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/84f6298b75d54f04a939f79f70894434, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/2695ce180caa4e68a39d3a6b40e25709, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/6745637e850a43f1ae0463d84078abe1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/e5b02b316f84465486c1a4d41832b7ce, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/8ebd59f4618c4fedb0379bc644f075cc, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/2cccb0f4958647be918e040091173531, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4e3867d26c324716a522732929ed47b1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/adf1516194914331acf01e7b77f867b8, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/c1b8c3370ad7424896b725be9bc1d275, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/02b687b02f5c4c23b60c1f57178c20a0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/e85d10329d024b51929b0e9fb0c394a6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/c42d6373e1db44d28ee87ad35cd14ce6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/1c61ce9185174b8395165f14858bdaf2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/8bcd54fc71bb45a3823a030401d2a9eb, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/d936dd94bf0a40e9a4f15e21613ff15f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/95dcc9fe111744dba6204d5e71117f8d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/7530f5b937cd401a8c6e3caf477853c1, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/824b76f7867647e781a3dcaf8879d56a, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/f5f8b01ec6cd4a4fada91962c850343c, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/2413b994d66b4c3597e592909fd714c4, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/44abe87c5766411dbb33020552637632, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/461a005cc02442fcbe813ac9c2ca77f2, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/fc5c2e22964a4f8f9c7c29a30fb9b2b3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4a97cdefdc7446e8aca8cf23ea7074f3, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/909cde13d21d4f6093ddbbc161009e3d, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/46e11776e752486cad90759243cecfa0, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/5210de496ead4f06a2c21ea541dcea09, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/e30a6bfb69a14376bb763a465bb32015, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/1d6e5175276b4b318d3eefb73df35f5f, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9410dcc62ac74ac18c25ad8d5e4a0ba6, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9183006fbfa446e48b16c2ac8e8475bb] to archive 2023-04-18 18:56:37,705 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 18:56:37,707 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9c17dc07c85c48bf82ec6b1816214534 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9c17dc07c85c48bf82ec6b1816214534 2023-04-18 18:56:37,708 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/c592bb64f5fc420aa49be513f17e4ef5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/c592bb64f5fc420aa49be513f17e4ef5 2023-04-18 18:56:37,709 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/5d1115953ab64140a00444ef9d0f7d68 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/5d1115953ab64140a00444ef9d0f7d68 2023-04-18 18:56:37,711 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4b03ad08bfeb43788d8d757c39d178f1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4b03ad08bfeb43788d8d757c39d178f1 2023-04-18 18:56:37,713 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9c330e9ad72b43648a8c79ddbb89f661 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9c330e9ad72b43648a8c79ddbb89f661 2023-04-18 18:56:37,714 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/24d197483e164a028e5c59c75d1b287c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/24d197483e164a028e5c59c75d1b287c 2023-04-18 18:56:37,715 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/65f50c4bf82149c7a23defc4fcc3b7c4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/65f50c4bf82149c7a23defc4fcc3b7c4 2023-04-18 18:56:37,716 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4079798575dd46b08e8e44e691d8b6ef to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4079798575dd46b08e8e44e691d8b6ef 2023-04-18 18:56:37,717 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/f14ee3a998be4ca89f736e1fa6adde27 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/f14ee3a998be4ca89f736e1fa6adde27 2023-04-18 18:56:37,718 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/241323bdc5704938bc1c0bda5b86db8a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/241323bdc5704938bc1c0bda5b86db8a 2023-04-18 18:56:37,719 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/b2d9d644ae7e4b6a8820dc348d1c173a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/b2d9d644ae7e4b6a8820dc348d1c173a 2023-04-18 18:56:37,720 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/78a7ebbe6df749199cf84f04c9cccac3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/78a7ebbe6df749199cf84f04c9cccac3 2023-04-18 18:56:37,721 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/527cdd887eb4479e90d8243bda973530 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/527cdd887eb4479e90d8243bda973530 2023-04-18 18:56:37,722 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/70804a300c8243768d488a85bfc653f1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/70804a300c8243768d488a85bfc653f1 2023-04-18 18:56:37,723 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/84f6298b75d54f04a939f79f70894434 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/84f6298b75d54f04a939f79f70894434 2023-04-18 18:56:37,724 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/2695ce180caa4e68a39d3a6b40e25709 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/2695ce180caa4e68a39d3a6b40e25709 2023-04-18 18:56:37,725 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/6745637e850a43f1ae0463d84078abe1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/6745637e850a43f1ae0463d84078abe1 2023-04-18 18:56:37,726 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/e5b02b316f84465486c1a4d41832b7ce to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/e5b02b316f84465486c1a4d41832b7ce 2023-04-18 18:56:37,727 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/8ebd59f4618c4fedb0379bc644f075cc to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/8ebd59f4618c4fedb0379bc644f075cc 2023-04-18 18:56:37,728 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/2cccb0f4958647be918e040091173531 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/2cccb0f4958647be918e040091173531 2023-04-18 18:56:37,729 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4e3867d26c324716a522732929ed47b1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4e3867d26c324716a522732929ed47b1 2023-04-18 18:56:37,730 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/adf1516194914331acf01e7b77f867b8 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/adf1516194914331acf01e7b77f867b8 2023-04-18 18:56:37,731 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/c1b8c3370ad7424896b725be9bc1d275 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/c1b8c3370ad7424896b725be9bc1d275 2023-04-18 18:56:37,732 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/02b687b02f5c4c23b60c1f57178c20a0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/02b687b02f5c4c23b60c1f57178c20a0 2023-04-18 18:56:37,733 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/e85d10329d024b51929b0e9fb0c394a6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/e85d10329d024b51929b0e9fb0c394a6 2023-04-18 18:56:37,734 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/c42d6373e1db44d28ee87ad35cd14ce6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/c42d6373e1db44d28ee87ad35cd14ce6 2023-04-18 18:56:37,735 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/1c61ce9185174b8395165f14858bdaf2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/1c61ce9185174b8395165f14858bdaf2 2023-04-18 18:56:37,736 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/8bcd54fc71bb45a3823a030401d2a9eb to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/8bcd54fc71bb45a3823a030401d2a9eb 2023-04-18 18:56:37,737 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/d936dd94bf0a40e9a4f15e21613ff15f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/d936dd94bf0a40e9a4f15e21613ff15f 2023-04-18 18:56:37,738 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/95dcc9fe111744dba6204d5e71117f8d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/95dcc9fe111744dba6204d5e71117f8d 2023-04-18 18:56:37,738 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/7530f5b937cd401a8c6e3caf477853c1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/7530f5b937cd401a8c6e3caf477853c1 2023-04-18 18:56:37,739 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/824b76f7867647e781a3dcaf8879d56a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/824b76f7867647e781a3dcaf8879d56a 2023-04-18 18:56:37,740 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/f5f8b01ec6cd4a4fada91962c850343c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/f5f8b01ec6cd4a4fada91962c850343c 2023-04-18 18:56:37,741 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/2413b994d66b4c3597e592909fd714c4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/2413b994d66b4c3597e592909fd714c4 2023-04-18 18:56:37,742 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/44abe87c5766411dbb33020552637632 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/44abe87c5766411dbb33020552637632 2023-04-18 18:56:37,743 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/461a005cc02442fcbe813ac9c2ca77f2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/461a005cc02442fcbe813ac9c2ca77f2 2023-04-18 18:56:37,744 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/fc5c2e22964a4f8f9c7c29a30fb9b2b3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/fc5c2e22964a4f8f9c7c29a30fb9b2b3 2023-04-18 18:56:37,745 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4a97cdefdc7446e8aca8cf23ea7074f3 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/4a97cdefdc7446e8aca8cf23ea7074f3 2023-04-18 18:56:37,746 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/909cde13d21d4f6093ddbbc161009e3d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/909cde13d21d4f6093ddbbc161009e3d 2023-04-18 18:56:37,747 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/46e11776e752486cad90759243cecfa0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/46e11776e752486cad90759243cecfa0 2023-04-18 18:56:37,748 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/5210de496ead4f06a2c21ea541dcea09 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/5210de496ead4f06a2c21ea541dcea09 2023-04-18 18:56:37,749 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/e30a6bfb69a14376bb763a465bb32015 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/e30a6bfb69a14376bb763a465bb32015 2023-04-18 18:56:37,750 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/1d6e5175276b4b318d3eefb73df35f5f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/1d6e5175276b4b318d3eefb73df35f5f 2023-04-18 18:56:37,751 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9410dcc62ac74ac18c25ad8d5e4a0ba6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9410dcc62ac74ac18c25ad8d5e4a0ba6 2023-04-18 18:56:37,752 DEBUG [StoreCloser-TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9183006fbfa446e48b16c2ac8e8475bb to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/9183006fbfa446e48b16c2ac8e8475bb 2023-04-18 18:56:37,756 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/recovered.edits/693.seqid, newMaxSeqId=693, maxSeqId=4 2023-04-18 18:56:37,757 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd. 2023-04-18 18:56:37,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c48561c851d33e29da879cafb393a8fd: 2023-04-18 18:56:37,759 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:37,759 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=c48561c851d33e29da879cafb393a8fd, regionState=CLOSED 2023-04-18 18:56:37,759 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681844197759"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681844197759"}]},"ts":"1681844197759"} 2023-04-18 18:56:37,761 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-18 18:56:37,761 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure c48561c851d33e29da879cafb393a8fd, server=jenkins-hbase4.apache.org,34053,1681844018952 in 1.5680 sec 2023-04-18 18:56:37,762 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-18 18:56:37,762 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c48561c851d33e29da879cafb393a8fd, UNASSIGN in 1.5720 sec 2023-04-18 18:56:37,763 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681844197762"}]},"ts":"1681844197762"} 2023-04-18 18:56:37,763 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-18 18:56:37,765 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-18 18:56:37,766 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5810 sec 2023-04-18 18:56:38,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-18 18:56:38,291 INFO [Listener at localhost/36951] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-18 18:56:38,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-18 18:56:38,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:56:38,294 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:56:38,295 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:56:38,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-18 18:56:38,298 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:38,300 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A, FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B, FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C, FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/recovered.edits] 2023-04-18 18:56:38,302 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/6867aa88cc6a4ef1a2e1e86f715fea49 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/6867aa88cc6a4ef1a2e1e86f715fea49 2023-04-18 18:56:38,303 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/ab3f684803094a5a9a339fe1978ce44f to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/A/ab3f684803094a5a9a339fe1978ce44f 2023-04-18 18:56:38,306 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/05a2bb20fb8f4e2fb0e5b4742f8ff4a6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/05a2bb20fb8f4e2fb0e5b4742f8ff4a6 2023-04-18 18:56:38,307 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/33cdcd167ae24ffe969b4f60b21744ae to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/B/33cdcd167ae24ffe969b4f60b21744ae 2023-04-18 18:56:38,309 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/0307b89f1f254ba3a4d56e570e941561 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/0307b89f1f254ba3a4d56e570e941561 2023-04-18 18:56:38,310 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/8d87d99d82c6451c8fc77693aa2e60cb to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/C/8d87d99d82c6451c8fc77693aa2e60cb 2023-04-18 18:56:38,312 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/recovered.edits/693.seqid to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd/recovered.edits/693.seqid 2023-04-18 18:56:38,312 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/.tmp/data/default/TestAcidGuarantees/c48561c851d33e29da879cafb393a8fd 2023-04-18 18:56:38,312 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 18:56:38,312 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-18 18:56:38,313 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-18 18:56:38,315 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418044149bd08f04fc49211debc122ccb62 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418044149bd08f04fc49211debc122ccb62 2023-04-18 18:56:38,316 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041813904312c90149baaeb71cb34c77e280 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041813904312c90149baaeb71cb34c77e280 2023-04-18 18:56:38,317 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304181a405c55378d459d88546852ab059065 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304181a405c55378d459d88546852ab059065 2023-04-18 18:56:38,318 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304181d60ca05b17043c386915f43698745c2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304181d60ca05b17043c386915f43698745c2 2023-04-18 18:56:38,319 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418263e36bec82849489de162f57a4945e1 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418263e36bec82849489de162f57a4945e1 2023-04-18 18:56:38,320 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418302fb6778ba04279a86b07ebd82f733c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418302fb6778ba04279a86b07ebd82f733c 2023-04-18 18:56:38,321 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418387888d0a69e49dfba2bc36f23bb8f17 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418387888d0a69e49dfba2bc36f23bb8f17 2023-04-18 18:56:38,321 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041849dc21f71f6e465ebed1ea015cb0f5a5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041849dc21f71f6e465ebed1ea015cb0f5a5 2023-04-18 18:56:38,322 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041854ae8bf4d2714a40bf072ead383ae8a6 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041854ae8bf4d2714a40bf072ead383ae8a6 2023-04-18 18:56:38,323 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041855374420f14b4903bf30a24666dd6de5 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041855374420f14b4903bf30a24666dd6de5 2023-04-18 18:56:38,324 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418596f95aa4aa34a4381d1345d676e6cd9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418596f95aa4aa34a4381d1345d676e6cd9 2023-04-18 18:56:38,325 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304185b725b46278c40c9b4daf8d9935b5f24 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304185b725b46278c40c9b4daf8d9935b5f24 2023-04-18 18:56:38,325 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304185e1df17d591c4300a9a270e7b6d54511 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304185e1df17d591c4300a9a270e7b6d54511 2023-04-18 18:56:38,326 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041862a98af4d39f4112b859540f39a7627c to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041862a98af4d39f4112b859540f39a7627c 2023-04-18 18:56:38,327 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304186aa61604ed5f470991952de2930c4172 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304186aa61604ed5f470991952de2930c4172 2023-04-18 18:56:38,328 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304186c8f0fa98bd846a1b2efa160ee5c2a65 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304186c8f0fa98bd846a1b2efa160ee5c2a65 2023-04-18 18:56:38,328 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304187041fc2efc0448d1b1ea54815ada3135 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304187041fc2efc0448d1b1ea54815ada3135 2023-04-18 18:56:38,329 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304187cfc0d1a6a604d38b9a2f79b61cee114 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304187cfc0d1a6a604d38b9a2f79b61cee114 2023-04-18 18:56:38,330 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041886f73705f2cd415cbfe46085042da2d2 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041886f73705f2cd415cbfe46085042da2d2 2023-04-18 18:56:38,331 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304189073d42d6ab948358e636cfffc98bd75 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304189073d42d6ab948358e636cfffc98bd75 2023-04-18 18:56:38,332 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304189a85c9238ad149b88029e2dbda3df9b4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304189a85c9238ad149b88029e2dbda3df9b4 2023-04-18 18:56:38,332 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a0114d6386fd44cab2ce80c76f70db4d to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a0114d6386fd44cab2ce80c76f70db4d 2023-04-18 18:56:38,333 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a29d62a728964a629db081a72fc0bfd0 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a29d62a728964a629db081a72fc0bfd0 2023-04-18 18:56:38,334 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418b1d78548b2b04b09a796cb413f8691ca to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418b1d78548b2b04b09a796cb413f8691ca 2023-04-18 18:56:38,335 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418b92d7ab12ee1414594747ca7e8550255 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418b92d7ab12ee1414594747ca7e8550255 2023-04-18 18:56:38,336 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418c46faac594624f6eacd2edf4ae42b027 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418c46faac594624f6eacd2edf4ae42b027 2023-04-18 18:56:38,337 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418c9e74a5139144b68b5c38ea04e85a19e to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418c9e74a5139144b68b5c38ea04e85a19e 2023-04-18 18:56:38,337 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418df1b9554f8c844cba5fe0f93e4682dc4 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418df1b9554f8c844cba5fe0f93e4682dc4 2023-04-18 18:56:38,338 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418dfc3d4d4f067466c802fcc36e9f0ea5a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418dfc3d4d4f067466c802fcc36e9f0ea5a 2023-04-18 18:56:38,339 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e10bf052840047338dffa4870c839c34 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e10bf052840047338dffa4870c839c34 2023-04-18 18:56:38,340 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e453aae6e7de4b43b4234154036788df to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e453aae6e7de4b43b4234154036788df 2023-04-18 18:56:38,341 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e76292350b7e457a9516f80593cd78bf to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e76292350b7e457a9516f80593cd78bf 2023-04-18 18:56:38,342 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e9feef5f70b340a5bdbcc3e6cbbf655a to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e9feef5f70b340a5bdbcc3e6cbbf655a 2023-04-18 18:56:38,343 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418ecdbe72e318649968b2f02295b6b3e51 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418ecdbe72e318649968b2f02295b6b3e51 2023-04-18 18:56:38,344 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f129450d5b574acbacc6ee2707dbc7f9 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f129450d5b574acbacc6ee2707dbc7f9 2023-04-18 18:56:38,345 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f854db00e5e341d8bc87d8c77b135510 to hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f854db00e5e341d8bc87d8c77b135510 2023-04-18 18:56:38,346 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-18 18:56:38,347 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:56:38,349 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-18 18:56:38,350 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-18 18:56:38,351 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:56:38,351 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-18 18:56:38,351 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681844198351"}]},"ts":"9223372036854775807"} 2023-04-18 18:56:38,352 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-18 18:56:38,353 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => c48561c851d33e29da879cafb393a8fd, NAME => 'TestAcidGuarantees,,1681844169680.c48561c851d33e29da879cafb393a8fd.', STARTKEY => '', ENDKEY => ''}] 2023-04-18 18:56:38,353 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-18 18:56:38,353 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681844198353"}]},"ts":"9223372036854775807"} 2023-04-18 18:56:38,354 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-18 18:56:38,356 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 18:56:38,356 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 64 msec 2023-04-18 18:56:38,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39391] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-18 18:56:38,396 INFO [Listener at localhost/36951] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-18 18:56:38,410 INFO [Listener at localhost/36951] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=227 (was 228), OpenFileDescriptor=578 (was 610), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=442 (was 386) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 170) - ProcessCount LEAK? -, AvailableMemoryMB=7023 (was 6909) - AvailableMemoryMB LEAK? - 2023-04-18 18:56:38,410 INFO [Listener at localhost/36951] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-18 18:56:38,410 INFO [Listener at localhost/36951] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-18 18:56:38,411 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x461317a6 to 127.0.0.1:65091 2023-04-18 18:56:38,411 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:38,411 DEBUG [Listener at localhost/36951] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-18 18:56:38,411 DEBUG [Listener at localhost/36951] util.JVMClusterUtil(257): Found active master hash=1596708042, stopped=false 2023-04-18 18:56:38,412 INFO [Listener at localhost/36951] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,39391,1681844017730 2023-04-18 18:56:38,413 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-18 18:56:38,413 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-18 18:56:38,413 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-18 18:56:38,413 INFO [Listener at localhost/36951] procedure2.ProcedureExecutor(629): Stopping 2023-04-18 18:56:38,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-18 18:56:38,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-18 18:56:38,414 DEBUG [Listener at localhost/36951] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x68e6aadc to 127.0.0.1:65091 2023-04-18 18:56:38,414 DEBUG [Listener at localhost/36951] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:38,414 INFO [Listener at localhost/36951] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34053,1681844018952' ***** 2023-04-18 18:56:38,414 INFO [Listener at localhost/36951] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-18 18:56:38,415 INFO [RS:0;jenkins-hbase4:34053] regionserver.HeapMemoryManager(220): Stopping 2023-04-18 18:56:38,415 INFO [RS:0;jenkins-hbase4:34053] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-18 18:56:38,415 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-18 18:56:38,415 INFO [RS:0;jenkins-hbase4:34053] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-18 18:56:38,415 INFO [RS:0;jenkins-hbase4:34053] regionserver.HRegionServer(3304): Received CLOSE for 8ca45fd8bf2d7472adff2285bed4eeeb 2023-04-18 18:56:38,416 INFO [RS:0;jenkins-hbase4:34053] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:38,416 DEBUG [RS:0;jenkins-hbase4:34053] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ab7257f to 127.0.0.1:65091 2023-04-18 18:56:38,417 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 8ca45fd8bf2d7472adff2285bed4eeeb, disabling compactions & flushes 2023-04-18 18:56:38,417 DEBUG [RS:0;jenkins-hbase4:34053] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:38,417 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb. 2023-04-18 18:56:38,417 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb. 2023-04-18 18:56:38,417 INFO [RS:0;jenkins-hbase4:34053] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-18 18:56:38,417 INFO [RS:0;jenkins-hbase4:34053] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-18 18:56:38,417 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb. after waiting 0 ms 2023-04-18 18:56:38,417 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb. 2023-04-18 18:56:38,417 INFO [RS:0;jenkins-hbase4:34053] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-18 18:56:38,417 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 8ca45fd8bf2d7472adff2285bed4eeeb 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-18 18:56:38,417 INFO [RS:0;jenkins-hbase4:34053] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-18 18:56:38,417 INFO [RS:0;jenkins-hbase4:34053] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-18 18:56:38,417 DEBUG [RS:0;jenkins-hbase4:34053] regionserver.HRegionServer(1479): Online Regions={8ca45fd8bf2d7472adff2285bed4eeeb=hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb., 1588230740=hbase:meta,,1.1588230740} 2023-04-18 18:56:38,418 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-18 18:56:38,418 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-18 18:56:38,418 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-18 18:56:38,418 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-18 18:56:38,418 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-18 18:56:38,418 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-18 18:56:38,419 DEBUG [RS:0;jenkins-hbase4:34053] regionserver.HRegionServer(1505): Waiting on 1588230740, 8ca45fd8bf2d7472adff2285bed4eeeb 2023-04-18 18:56:38,426 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/namespace/8ca45fd8bf2d7472adff2285bed4eeeb/.tmp/info/c72ec11a2f9040c9b03fc6a733ee7244 2023-04-18 18:56:38,430 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/namespace/8ca45fd8bf2d7472adff2285bed4eeeb/.tmp/info/c72ec11a2f9040c9b03fc6a733ee7244 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/namespace/8ca45fd8bf2d7472adff2285bed4eeeb/info/c72ec11a2f9040c9b03fc6a733ee7244 2023-04-18 18:56:38,434 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/namespace/8ca45fd8bf2d7472adff2285bed4eeeb/info/c72ec11a2f9040c9b03fc6a733ee7244, entries=2, sequenceid=6, filesize=4.8 K 2023-04-18 18:56:38,434 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 8ca45fd8bf2d7472adff2285bed4eeeb in 17ms, sequenceid=6, compaction requested=false 2023-04-18 18:56:38,439 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/namespace/8ca45fd8bf2d7472adff2285bed4eeeb/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-18 18:56:38,440 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb. 2023-04-18 18:56:38,440 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 8ca45fd8bf2d7472adff2285bed4eeeb: 2023-04-18 18:56:38,440 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1681844020960.8ca45fd8bf2d7472adff2285bed4eeeb. 2023-04-18 18:56:38,489 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-18 18:56:38,620 DEBUG [RS:0;jenkins-hbase4:34053] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-18 18:56:38,820 DEBUG [RS:0;jenkins-hbase4:34053] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-18 18:56:38,826 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/.tmp/info/b7d63b683989420c975f7e53aa3acff8 2023-04-18 18:56:38,830 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b7d63b683989420c975f7e53aa3acff8 2023-04-18 18:56:38,835 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/.tmp/rep_barrier/7e1b9fe30c4d499e8b8b9b8024234969 2023-04-18 18:56:38,838 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7e1b9fe30c4d499e8b8b9b8024234969 2023-04-18 18:56:39,020 DEBUG [RS:0;jenkins-hbase4:34053] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-18 18:56:39,221 DEBUG [RS:0;jenkins-hbase4:34053] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-18 18:56:39,243 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/.tmp/table/8579ddfca3d94d7f905c652a3fe0d29a 2023-04-18 18:56:39,246 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8579ddfca3d94d7f905c652a3fe0d29a 2023-04-18 18:56:39,246 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/.tmp/info/b7d63b683989420c975f7e53aa3acff8 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/info/b7d63b683989420c975f7e53aa3acff8 2023-04-18 18:56:39,249 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b7d63b683989420c975f7e53aa3acff8 2023-04-18 18:56:39,249 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/info/b7d63b683989420c975f7e53aa3acff8, entries=22, sequenceid=93, filesize=7.1 K 2023-04-18 18:56:39,251 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/.tmp/rep_barrier/7e1b9fe30c4d499e8b8b9b8024234969 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/rep_barrier/7e1b9fe30c4d499e8b8b9b8024234969 2023-04-18 18:56:39,253 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7e1b9fe30c4d499e8b8b9b8024234969 2023-04-18 18:56:39,253 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/rep_barrier/7e1b9fe30c4d499e8b8b9b8024234969, entries=6, sequenceid=93, filesize=5.5 K 2023-04-18 18:56:39,254 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/.tmp/table/8579ddfca3d94d7f905c652a3fe0d29a as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/table/8579ddfca3d94d7f905c652a3fe0d29a 2023-04-18 18:56:39,257 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8579ddfca3d94d7f905c652a3fe0d29a 2023-04-18 18:56:39,257 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/table/8579ddfca3d94d7f905c652a3fe0d29a, entries=9, sequenceid=93, filesize=5.5 K 2023-04-18 18:56:39,257 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 839ms, sequenceid=93, compaction requested=false 2023-04-18 18:56:39,264 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-18 18:56:39,265 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-18 18:56:39,266 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-18 18:56:39,266 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-18 18:56:39,266 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-18 18:56:39,284 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-18 18:56:39,285 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-18 18:56:39,421 INFO [RS:0;jenkins-hbase4:34053] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34053,1681844018952; all regions closed. 2023-04-18 18:56:39,429 DEBUG [RS:0;jenkins-hbase4:34053] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/oldWALs 2023-04-18 18:56:39,429 INFO [RS:0;jenkins-hbase4:34053] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34053%2C1681844018952.meta:.meta(num 1681844020656) 2023-04-18 18:56:39,434 DEBUG [RS:0;jenkins-hbase4:34053] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/oldWALs 2023-04-18 18:56:39,434 INFO [RS:0;jenkins-hbase4:34053] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34053%2C1681844018952:(num 1681844020442) 2023-04-18 18:56:39,434 DEBUG [RS:0;jenkins-hbase4:34053] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:39,434 INFO [RS:0;jenkins-hbase4:34053] regionserver.LeaseManager(133): Closed leases 2023-04-18 18:56:39,435 INFO [RS:0;jenkins-hbase4:34053] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-18 18:56:39,435 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-18 18:56:39,436 INFO [RS:0;jenkins-hbase4:34053] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34053 2023-04-18 18:56:39,442 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34053,1681844018952 2023-04-18 18:56:39,442 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-18 18:56:39,442 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-18 18:56:39,442 ERROR [Listener at localhost/36951-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3e969770 rejected from java.util.concurrent.ThreadPoolExecutor@3aa9047a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2335] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-18 18:56:39,444 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,34053,1681844018952] 2023-04-18 18:56:39,445 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,34053,1681844018952; numProcessing=1 2023-04-18 18:56:39,446 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,34053,1681844018952 already deleted, retry=false 2023-04-18 18:56:39,446 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,34053,1681844018952 expired; onlineServers=0 2023-04-18 18:56:39,446 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39391,1681844017730' ***** 2023-04-18 18:56:39,446 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-18 18:56:39,446 DEBUG [M:0;jenkins-hbase4:39391] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e16def1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-18 18:56:39,446 INFO [M:0;jenkins-hbase4:39391] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39391,1681844017730 2023-04-18 18:56:39,446 INFO [M:0;jenkins-hbase4:39391] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39391,1681844017730; all regions closed. 2023-04-18 18:56:39,446 DEBUG [M:0;jenkins-hbase4:39391] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 18:56:39,446 DEBUG [M:0;jenkins-hbase4:39391] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-18 18:56:39,446 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-18 18:56:39,446 DEBUG [M:0;jenkins-hbase4:39391] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-18 18:56:39,446 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681844020144] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681844020144,5,FailOnTimeoutGroup] 2023-04-18 18:56:39,446 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681844020147] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681844020147,5,FailOnTimeoutGroup] 2023-04-18 18:56:39,446 INFO [M:0;jenkins-hbase4:39391] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-18 18:56:39,447 INFO [M:0;jenkins-hbase4:39391] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-18 18:56:39,447 INFO [M:0;jenkins-hbase4:39391] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-18 18:56:39,447 DEBUG [M:0;jenkins-hbase4:39391] master.HMaster(1512): Stopping service threads 2023-04-18 18:56:39,447 INFO [M:0;jenkins-hbase4:39391] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-18 18:56:39,447 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-18 18:56:39,447 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-18 18:56:39,447 ERROR [M:0;jenkins-hbase4:39391] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-18 18:56:39,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-18 18:56:39,448 INFO [M:0;jenkins-hbase4:39391] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-18 18:56:39,448 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-18 18:56:39,448 DEBUG [M:0;jenkins-hbase4:39391] zookeeper.ZKUtil(398): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-18 18:56:39,448 WARN [M:0;jenkins-hbase4:39391] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-18 18:56:39,449 INFO [M:0;jenkins-hbase4:39391] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-18 18:56:39,449 INFO [M:0;jenkins-hbase4:39391] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-18 18:56:39,449 DEBUG [M:0;jenkins-hbase4:39391] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-18 18:56:39,449 INFO [M:0;jenkins-hbase4:39391] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-18 18:56:39,449 DEBUG [M:0;jenkins-hbase4:39391] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-18 18:56:39,449 DEBUG [M:0;jenkins-hbase4:39391] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-18 18:56:39,449 DEBUG [M:0;jenkins-hbase4:39391] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-18 18:56:39,449 INFO [M:0;jenkins-hbase4:39391] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.68 KB heapSize=393 KB 2023-04-18 18:56:39,545 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-18 18:56:39,545 INFO [RS:0;jenkins-hbase4:34053] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34053,1681844018952; zookeeper connection closed. 2023-04-18 18:56:39,545 ERROR [Listener at localhost/36951-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4439948b rejected from java.util.concurrent.ThreadPoolExecutor@3aa9047a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2335] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-18 18:56:39,546 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): regionserver:34053-0x101989f00970001, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-18 18:56:39,546 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3c8175ff] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3c8175ff 2023-04-18 18:56:39,546 ERROR [Listener at localhost/36951-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3c66f106 rejected from java.util.concurrent.ThreadPoolExecutor@3aa9047a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2335] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-18 18:56:39,547 INFO [Listener at localhost/36951] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-18 18:56:39,862 INFO [M:0;jenkins-hbase4:39391] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.68 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/957d9fe8dec344aeb3a700c1383e71f1 2023-04-18 18:56:39,865 INFO [M:0;jenkins-hbase4:39391] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 957d9fe8dec344aeb3a700c1383e71f1 2023-04-18 18:56:39,866 DEBUG [M:0;jenkins-hbase4:39391] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/957d9fe8dec344aeb3a700c1383e71f1 as hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/957d9fe8dec344aeb3a700c1383e71f1 2023-04-18 18:56:39,869 INFO [M:0;jenkins-hbase4:39391] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 957d9fe8dec344aeb3a700c1383e71f1 2023-04-18 18:56:39,870 INFO [M:0;jenkins-hbase4:39391] regionserver.HStore(1080): Added hdfs://localhost:40835/user/jenkins/test-data/3e7d2145-f65d-25fe-0af6-8d9e9665c6ab/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/957d9fe8dec344aeb3a700c1383e71f1, entries=65, sequenceid=573, filesize=24.7 K 2023-04-18 18:56:39,870 INFO [M:0;jenkins-hbase4:39391] regionserver.HRegion(2948): Finished flush of dataSize ~344.68 KB/352954, heapSize ~392.98 KB/402416, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 421ms, sequenceid=573, compaction requested=false 2023-04-18 18:56:39,872 INFO [M:0;jenkins-hbase4:39391] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-18 18:56:39,872 DEBUG [M:0;jenkins-hbase4:39391] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-18 18:56:39,874 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-18 18:56:39,874 INFO [M:0;jenkins-hbase4:39391] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-18 18:56:39,875 INFO [M:0;jenkins-hbase4:39391] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39391 2023-04-18 18:56:39,878 DEBUG [M:0;jenkins-hbase4:39391] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,39391,1681844017730 already deleted, retry=false 2023-04-18 18:56:39,980 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-18 18:56:39,980 INFO [M:0;jenkins-hbase4:39391] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39391,1681844017730; zookeeper connection closed. 2023-04-18 18:56:39,980 ERROR [Listener at localhost/36951-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2c79b001 rejected from java.util.concurrent.ThreadPoolExecutor@1723ed0f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3908] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-18 18:56:39,981 DEBUG [Listener at localhost/36951-EventThread] zookeeper.ZKWatcher(600): master:39391-0x101989f00970000, quorum=127.0.0.1:65091, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-18 18:56:39,981 ERROR [Listener at localhost/36951-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@657cb60f rejected from java.util.concurrent.ThreadPoolExecutor@1723ed0f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3908] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-18 18:56:39,982 WARN [Listener at localhost/36951] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-18 18:56:39,985 INFO [Listener at localhost/36951] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-18 18:56:40,095 WARN [BP-1482039454-172.31.14.131-1681844014371 heartbeating to localhost/127.0.0.1:40835] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-18 18:56:40,095 WARN [BP-1482039454-172.31.14.131-1681844014371 heartbeating to localhost/127.0.0.1:40835] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1482039454-172.31.14.131-1681844014371 (Datanode Uuid 309c0e06-eb1a-47ea-b39b-d28d4ae1a8b8) service to localhost/127.0.0.1:40835 2023-04-18 18:56:40,098 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/cluster_a8fdfc0f-6800-c939-7c96-ea4c4c9b2e3f/dfs/data/data1/current/BP-1482039454-172.31.14.131-1681844014371] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-18 18:56:40,099 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f4cb6f4a-8beb-c1c0-444d-3afc191254fc/cluster_a8fdfc0f-6800-c939-7c96-ea4c4c9b2e3f/dfs/data/data2/current/BP-1482039454-172.31.14.131-1681844014371] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-18 18:56:40,138 INFO [Listener at localhost/36951] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-18 18:56:40,326 INFO [Listener at localhost/36951] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-18 18:56:40,517 INFO [Listener at localhost/36951] hbase.HBaseTestingUtility(1293): Minicluster is down